2023-09-30T00:30:17.608 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:30:17.608 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764 --verbose --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/orig.config.yaml 2023-09-30T00:30:17.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:30:17.741 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} email: yweinste@redhat.com first_in_suite: false job_id: '7406764' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd_async_recovery_min_cost: 1 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd inject bad map crc probability: 0.1 osd max markdown count: 1000 osd max scrubs: 2 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - must scrub before tier agent can activate sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZlaQEP+7adkueMb/+ERFQy3Dwhho1Wb12ZhXccY71Dnj1nVL4Fj5qVr8Ipasufq3xp01VCo5NHXPqmMBXL7Sc= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJJ2MZTCXYCd/ZJ7xqPR+owGRouh0wJqcJ14wyaj2b535kQMDTxSikwVVcIMqPOdR3lKn5wcwb3bg42m1dINiU= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - rados: balance_reads: true clients: - client.0 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:30:17.742 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:30:17.742 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:30:17.742 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:30:17.744 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:30:17.764 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:30:17.765 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:30:17.765 INFO:teuthology.packaging:ref: None 2023-09-30T00:30:17.765 INFO:teuthology.packaging:tag: None 2023-09-30T00:30:17.765 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:30:17.765 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:30:17.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:30:17.984 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:30:17.984 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:30:17.990 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:30:17.990 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:30:18.004 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:30:18.020 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:30:18.025 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:30:18.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:25:21.443005', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZlaQEP+7adkueMb/+ERFQy3Dwhho1Wb12ZhXccY71Dnj1nVL4Fj5qVr8Ipasufq3xp01VCo5NHXPqmMBXL7Sc='} 2023-09-30T00:30:18.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:25:21.444559', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJJ2MZTCXYCd/ZJ7xqPR+owGRouh0wJqcJ14wyaj2b535kQMDTxSikwVVcIMqPOdR3lKn5wcwb3bg42m1dINiU='} 2023-09-30T00:30:18.078 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:30:18.083 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T00:30:18.083 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T00:30:18.083 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:30:18.157 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f82894fbf70>, signals=[15]) 2023-09-30T00:30:18.158 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:30:18.165 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:30:18.165 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:30:18.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:30:18.235 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:30:18.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:30:18.307 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:30:18.314 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2023-09-30T00:30:18.332 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2023-09-30T00:30:18.333 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2023-09-30T00:30:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T00:30:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T00:30:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:ID="rhel" 2023-09-30T00:30:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="fedora" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="8.4" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T00:30:18.391 INFO:teuthology.orchestra.run.smithi017.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T00:30:18.392 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:30:18.392 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:30:18.392 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:30:18.392 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T00:30:18.392 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T00:30:18.392 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T00:30:18.393 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2023-09-30T00:30:18.414 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2023-09-30T00:30:18.431 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2023-09-30T00:30:18.432 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2023-09-30T00:30:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T00:30:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T00:30:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:ID="rhel" 2023-09-30T00:30:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="fedora" 2023-09-30T00:30:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8.4" 2023-09-30T00:30:18.492 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T00:30:18.493 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T00:30:18.494 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T00:30:18.494 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-09-30T00:30:18.514 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:30:18.523 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:30:18.531 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:30:18.531 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:30:18.533 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:30:18.551 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:30:18.557 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:30:18.557 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:30:18.590 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:30:18.605 INFO:teuthology.orchestra.run.smithi017.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T00:30:18.610 INFO:teuthology.orchestra.run.smithi071.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T00:30:18.611 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:30:18.664 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:30:18.678 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:30:18.678 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:30:18.679 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:30:18.679 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:30:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:30:18.695 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-09-30T00:30:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:30:18.696 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2023-09-30T00:30:18.753 INFO:teuthology.orchestra.run.smithi017.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:30:18.753 INFO:teuthology.task.kernel:Running kernel on smithi017: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:30:18.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2023-09-30T00:30:18.754 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:30:18.754 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:30:18.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2023-09-30T00:30:20.185 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:30:20.185 INFO:teuthology.orchestra.run.smithi017.stdout:Unable to read consumer identity 2023-09-30T00:30:20.225 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:30:20.225 INFO:teuthology.orchestra.run.smithi071.stdout:Unable to read consumer identity 2023-09-30T00:30:21.698 INFO:teuthology.orchestra.run.smithi017.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2023-09-30T00:30:21.707 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-09-30T00:30:22.311 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 28 MB/s | 16 MB 00:00 2023-09-30T00:30:22.315 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 28 MB/s | 16 MB 00:00 2023-09-30T00:30:25.596 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-09-30T00:30:25.781 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-09-30T00:30:26.324 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T00:30:26.325 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T00:30:26.325 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T00:30:26.478 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:30:26.479 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:30:26.479 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:30:26.569 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T00:30:26.569 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T00:30:26.569 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T00:30:26.587 DEBUG:teuthology.orchestra.run.smithi071:> echo no | sudo yum reinstall kernel || true 2023-09-30T00:30:26.733 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:30:26.734 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:30:26.734 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:30:26.833 DEBUG:teuthology.orchestra.run.smithi017:> echo no | sudo yum reinstall kernel || true 2023-09-30T00:30:26.929 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:30:26.930 INFO:teuthology.orchestra.run.smithi071.stdout:Unable to read consumer identity 2023-09-30T00:30:27.198 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:30:27.198 INFO:teuthology.orchestra.run.smithi017.stdout:Unable to read consumer identity 2023-09-30T00:30:27.215 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 12:30:25 AM UTC. 2023-09-30T00:30:27.338 INFO:teuthology.orchestra.run.smithi071.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:30:27.338 INFO:teuthology.orchestra.run.smithi071.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:27.338 INFO:teuthology.orchestra.run.smithi071.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:27.368 INFO:teuthology.orchestra.run.smithi071.stderr:Error: No packages marked for reinstall. 2023-09-30T00:30:27.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum reinstall -y kernel || true 2023-09-30T00:30:27.502 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 12:30:25 AM UTC. 2023-09-30T00:30:27.630 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:30:27.630 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:27.630 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:27.649 INFO:teuthology.orchestra.run.smithi017.stderr:Error: No packages marked for reinstall. 2023-09-30T00:30:27.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum reinstall -y kernel || true 2023-09-30T00:30:27.762 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:30:27.762 INFO:teuthology.orchestra.run.smithi071.stdout:Unable to read consumer identity 2023-09-30T00:30:28.035 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:03 ago on Sat 30 Sep 2023 12:30:25 AM UTC. 2023-09-30T00:30:28.066 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:30:28.067 INFO:teuthology.orchestra.run.smithi017.stdout:Unable to read consumer identity 2023-09-30T00:30:28.170 INFO:teuthology.orchestra.run.smithi071.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:30:28.170 INFO:teuthology.orchestra.run.smithi071.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:28.171 INFO:teuthology.orchestra.run.smithi071.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:28.187 INFO:teuthology.orchestra.run.smithi071.stderr:Error: No packages marked for reinstall. 2023-09-30T00:30:28.235 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:30:28.359 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.360 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.360 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:30:28.360 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:30:28.360 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T00:30:28.360 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:30:28.360 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:30:28.361 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:03 ago on Sat 30 Sep 2023 12:30:25 AM UTC. 2023-09-30T00:30:28.484 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.485 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2023-09-30T00:30:28.511 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2023-09-30T00:30:28.511 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.02 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 99.el8_4.1 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Group : System Environment/Base 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 9264631 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T00:30:28.512 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Relocations : (not relocatable) 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : Red Hat, Inc. 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : Red Hat, Inc. 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:30:28.513 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:30:28.514 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:30:28.514 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:30:28.514 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2023-09-30T00:30:28.514 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:30:28.514 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:30:28.515 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:30:28.515 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:30:28.646 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:30:28.647 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:28.647 INFO:teuthology.orchestra.run.smithi017.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:30:28.659 INFO:teuthology.orchestra.run.smithi017.stderr:Error: No packages marked for reinstall. 2023-09-30T00:30:28.716 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:30:28.847 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.848 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.848 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.848 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:30:28.848 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:30:28.848 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:30:28.848 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:30:28.848 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:30:28.979 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.979 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:28.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.02 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 99.el8_4.1 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T00:30:29.008 INFO:teuthology.orchestra.run.smithi017.stdout:Group : System Environment/Base 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 9264631 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Relocations : (not relocatable) 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : Red Hat, Inc. 2023-09-30T00:30:29.009 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : Red Hat, Inc. 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:30:29.010 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:30:29.011 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2023-09-30T00:30:29.011 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:30:29.011 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:30:29.012 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:30:29.012 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:29.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:30:29.882 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2023-09-30T00:30:30.304 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2023-09-30T00:30:32.048 INFO:teuthology.orchestra.run.smithi071.stderr:done 2023-09-30T00:30:32.051 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-09-30T00:30:32.064 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.y3qSK0JXi3 2023-09-30T00:30:32.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y3qSK0JXi3 2023-09-30T00:30:32.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.y3qSK0JXi3 2023-09-30T00:30:32.145 INFO:teuthology.orchestra.run.smithi017.stderr:done 2023-09-30T00:30:32.147 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2023-09-30T00:30:32.209 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.LDrE5Lcbzv 2023-09-30T00:30:32.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LDrE5Lcbzv 2023-09-30T00:30:32.289 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.y3qSK0JXi3 is 6KB 2023-09-30T00:30:32.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.LDrE5Lcbzv 2023-09-30T00:30:32.337 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.y3qSK0JXi3 2023-09-30T00:30:32.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:30:32.420 INFO:teuthology.orchestra.run.smithi071.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T00:30:32.420 INFO:teuthology.orchestra.run.smithi071.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T00:30:32.420 INFO:teuthology.orchestra.run.smithi071.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T00:30:32.420 INFO:teuthology.orchestra.run.smithi071.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T00:30:32.421 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:32.445 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.LDrE5Lcbzv is 6KB 2023-09-30T00:30:32.493 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.LDrE5Lcbzv 2023-09-30T00:30:32.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:30:32.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2023-09-30T00:30:32.576 INFO:teuthology.orchestra.run.smithi017.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T00:30:32.576 INFO:teuthology.orchestra.run.smithi017.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T00:30:32.576 INFO:teuthology.orchestra.run.smithi017.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T00:30:32.577 INFO:teuthology.orchestra.run.smithi017.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T00:30:32.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:30:32.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2023-09-30T00:31:02.606 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T00:31:02.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:31:02.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:31:02.752 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:31:02.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-09-30T00:31:02.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:31:20.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-09-30T00:31:21.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2023-09-30T00:31:29.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:31:29.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:31:30.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-09-30T00:31:30.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:31:33.018 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-09-30T00:31:45.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:31:45.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:32:03.457 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-09-30T00:32:18.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:32:18.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:32:19.266 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-09-30T00:32:20.562 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:32:20.562 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T00:32:20.563 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-09-30T00:32:20.943 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T00:32:20.943 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:32:20.943 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:32:20.943 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T00:32:21.945 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T00:32:21.945 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:32:22.184 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2023-09-30T00:32:22.197 DEBUG:teuthology.parallel:result is None 2023-09-30T00:32:30.215 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:32:42.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-09-30T00:32:42.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:32:42.555 DEBUG:teuthology.orchestra.run.smithi017:> true 2023-09-30T00:32:42.849 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-09-30T00:32:42.849 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T00:32:42.850 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2023-09-30T00:32:42.866 INFO:teuthology.orchestra.run.smithi017.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T00:32:42.866 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:32:42.866 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:32:42.867 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T00:32:43.867 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:32:43.868 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:32:43.948 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2023-09-30T00:32:43.965 DEBUG:teuthology.parallel:result is None 2023-09-30T00:32:43.965 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:32:43.973 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:32:43.973 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:32:43.975 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:32:43.993 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:32:44.000 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:32:44.007 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:32:44.007 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:32:44.032 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:32:44.066 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:32:44.073 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:32:44.074 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:32:44.092 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:32:44.130 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:32:44.141 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:32:44.143 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:32:44.155 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:32:44.156 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:32:44.163 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:32:44.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:32:44.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:32:44.225 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:32:44.234 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:32:44.234 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:32:44.255 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:32:44.281 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:32:44.354 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:32:44.394 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:32:44.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:32:44.518 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:32:44.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:32:44.584 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:32:44.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:32:44.671 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:32:44.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:32:44.795 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:32:44.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:32:44.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2023-09-30T00:32:44.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2023-09-30T00:32:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:32:45.026 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:32:45.248 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:32:45.257 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:32:45.257 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:32:45.267 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:32:45.275 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:32:45.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2023-09-30T00:32:45.332 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: [ OK ] 2023-09-30T00:32:45.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2023-09-30T00:32:45.394 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: [ OK ] 2023-09-30T00:32:45.396 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:32:45.396 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2023-09-30T00:32:45.420 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2023-09-30T00:32:45.420 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2023-09-30T00:32:45.442 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2023-09-30T00:32:45.442 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:32:45.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:32:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:45.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:32:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:45.513 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:32:45.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:32:45.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:32:45.593 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:32:45.602 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-30T00:32:45.608 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-30T00:32:45.731 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:32:45.746 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:32:45.747 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:40:58.296 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2023-09-30T00:40:58.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-09-30T00:40:58.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:40:58.368 DEBUG:teuthology.orchestra.run.smithi017:> true 2023-09-30T00:40:58.434 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2023-09-30T00:40:58.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:40:58.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:40:58.505 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-09-30T00:40:58.570 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-09-30T00:40:58.570 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:40:58.588 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:40:58.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:40:58.588 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:40:58.590 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:40:58.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:40:58.619 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:40:58.635 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:40:58.638 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:40:58.653 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:40:58.670 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2023-09-30T00:40:58.681 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2023-09-30T00:40:58.696 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:40:58.696 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2023-09-30T00:40:58.706 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2023-09-30T00:40:58.710 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:40:58.721 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:40:58.734 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:40:58.779 INFO:teuthology.orchestra.run.smithi017.stderr:bash: ntpq: command not found 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:210 Number of sources = 4 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.782 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.800 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2023-09-30T00:40:58.803 INFO:teuthology.orchestra.run.smithi071.stdout:210 Number of sources = 4 2023-09-30T00:40:58.803 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:40:58.804 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-09-30T00:40:58.804 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.804 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.804 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:40:58.805 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:40:58.813 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:40:58.813 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:40:58.813 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:40:58.814 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:40:58.819 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:40:58.819 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:40:58.820 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:40:58.820 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:40:58.822 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:40:58.822 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:40:58.970 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:40:58.970 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:40:59.045 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:40:59.045 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:40:59.247 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:40:59.248 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:40:59.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:40:59.273 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:40:59.274 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:40:59.323 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:40:59.324 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:40:59.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:40:59.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:40:59.352 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:40:59.353 DEBUG:teuthology.orchestra.run.smithi017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:40:59.413 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:40:59.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:40:59.477 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2023-09-30T00:40:59.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2023-09-30T00:40:59.499 DEBUG:teuthology.orchestra.run.smithi017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:40:59.565 INFO:teuthology.orchestra.run.smithi017.stdout:check_obsoletes = 1 2023-09-30T00:40:59.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum clean all 2023-09-30T00:40:59.867 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:40:59.978 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:41:00.282 INFO:teuthology.orchestra.run.smithi071.stdout:44 files removed 2023-09-30T00:41:00.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2023-09-30T00:41:00.378 INFO:teuthology.orchestra.run.smithi017.stdout:44 files removed 2023-09-30T00:41:00.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-radosgw 2023-09-30T00:41:00.681 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:41:00.799 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:41:01.798 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 116 kB/s | 78 kB 00:00 2023-09-30T00:41:01.927 INFO:teuthology.orchestra.run.smithi017.stdout:ceph packages for x86_64 115 kB/s | 78 kB 00:00 2023-09-30T00:41:02.536 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 13 kB/s | 9.2 kB 00:00 2023-09-30T00:41:02.621 INFO:teuthology.orchestra.run.smithi017.stdout:ceph noarch packages 14 kB/s | 9.2 kB 00:00 2023-09-30T00:41:03.125 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2023-09-30T00:41:03.234 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-09-30T00:41:03.253 INFO:teuthology.orchestra.run.smithi017.stdout:ceph source packages 2.4 kB/s | 1.5 kB 00:00 2023-09-30T00:41:03.370 INFO:teuthology.orchestra.run.smithi017.stdout:Copr repo for python3-asyncssh owned by ceph 41 kB/s | 3.5 kB 00:00 2023-09-30T00:41:04.010 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-09-30T00:41:05.542 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 16 MB 00:02 2023-09-30T00:41:07.556 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 4.9 MB/s | 24 kB 00:00 2023-09-30T00:41:07.957 INFO:teuthology.orchestra.run.smithi017.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 104 MB/s | 33 MB 00:00 2023-09-30T00:41:08.779 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-09-30T00:41:09.179 INFO:teuthology.orchestra.run.smithi071.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 120 MB/s | 38 MB 00:00 2023-09-30T00:41:13.892 INFO:teuthology.orchestra.run.smithi017.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 106 MB/s | 38 MB 00:00 2023-09-30T00:41:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-09-30T00:41:22.942 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout:==================================================================================================== 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout:==================================================================================================== 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:41:22.945 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T00:41:22.946 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:41:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:==================================================================================================== 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:Install 21 Packages 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:41:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 63 M 2023-09-30T00:41:22.949 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:41:23.648 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:41:23.651 INFO:teuthology.orchestra.run.smithi017.stdout:==================================================================================================== 2023-09-30T00:41:23.652 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:41:23.652 INFO:teuthology.orchestra.run.smithi017.stdout:==================================================================================================== 2023-09-30T00:41:23.652 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:41:23.652 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:41:23.652 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrading: 2023-09-30T00:41:23.653 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:41:23.653 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:41:23.653 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:41:23.653 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:41:23.653 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:41:23.653 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:41:23.654 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:41:23.654 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:41:23.654 INFO:teuthology.orchestra.run.smithi017.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:41:23.654 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T00:41:23.654 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:41:23.654 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T00:41:23.655 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:41:23.655 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:41:23.655 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T00:41:23.655 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:41:23.655 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:41:23.655 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:41:23.656 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T00:41:23.656 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:41:23.656 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:41:23.656 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:41:23.656 INFO:teuthology.orchestra.run.smithi017.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:41:23.656 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:41:23.657 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:41:23.657 INFO:teuthology.orchestra.run.smithi017.stdout:==================================================================================================== 2023-09-30T00:41:23.657 INFO:teuthology.orchestra.run.smithi017.stdout:Install 21 Packages 2023-09-30T00:41:23.657 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrade 2 Packages 2023-09-30T00:41:23.657 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:41:23.658 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 63 M 2023-09-30T00:41:23.658 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:41:25.187 INFO:teuthology.orchestra.run.smithi071.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 2.4 MB/s | 5.2 MB 00:02 2023-09-30T00:41:25.256 INFO:teuthology.orchestra.run.smithi071.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 358 kB/s | 24 kB 00:00 2023-09-30T00:41:25.537 INFO:teuthology.orchestra.run.smithi071.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 2.8 MB/s | 796 kB 00:00 2023-09-30T00:41:25.702 INFO:teuthology.orchestra.run.smithi071.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 3.0 MB/s | 506 kB 00:00 2023-09-30T00:41:27.164 INFO:teuthology.orchestra.run.smithi017.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.5 MB/s | 5.2 MB 00:03 2023-09-30T00:41:27.237 INFO:teuthology.orchestra.run.smithi017.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 332 kB/s | 24 kB 00:00 2023-09-30T00:41:27.384 INFO:teuthology.orchestra.run.smithi017.stdout:(3/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 4.5 MB/s | 17 MB 00:03 2023-09-30T00:41:27.410 INFO:teuthology.orchestra.run.smithi071.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 3.3 MB/s | 5.6 MB 00:01 2023-09-30T00:41:27.486 INFO:teuthology.orchestra.run.smithi071.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 593 kB/s | 45 kB 00:00 2023-09-30T00:41:27.504 INFO:teuthology.orchestra.run.smithi017.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 4.1 MB/s | 506 kB 00:00 2023-09-30T00:41:27.558 INFO:teuthology.orchestra.run.smithi071.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.6 MB/s | 115 kB 00:00 2023-09-30T00:41:27.619 INFO:teuthology.orchestra.run.smithi017.stdout:(5/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 2.0 MB/s | 796 kB 00:00 2023-09-30T00:41:27.638 INFO:teuthology.orchestra.run.smithi071.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.3 MB/s | 190 kB 00:00 2023-09-30T00:41:27.697 INFO:teuthology.orchestra.run.smithi017.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 577 kB/s | 45 kB 00:00 2023-09-30T00:41:27.784 INFO:teuthology.orchestra.run.smithi017.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.3 MB/s | 115 kB 00:00 2023-09-30T00:41:27.800 INFO:teuthology.orchestra.run.smithi071.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 2.2 MB/s | 370 kB 00:00 2023-09-30T00:41:27.951 INFO:teuthology.orchestra.run.smithi017.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.1 MB/s | 190 kB 00:00 2023-09-30T00:41:27.970 INFO:teuthology.orchestra.run.smithi071.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 2.1 MB/s | 358 kB 00:00 2023-09-30T00:41:28.061 INFO:teuthology.orchestra.run.smithi071.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.2 MB/s | 107 kB 00:00 2023-09-30T00:41:28.203 INFO:teuthology.orchestra.run.smithi017.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 1.4 MB/s | 370 kB 00:00 2023-09-30T00:41:28.390 INFO:teuthology.orchestra.run.smithi017.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 1.9 MB/s | 358 kB 00:00 2023-09-30T00:41:28.476 INFO:teuthology.orchestra.run.smithi017.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.2 MB/s | 107 kB 00:00 2023-09-30T00:41:28.565 INFO:teuthology.orchestra.run.smithi017.stdout:(12/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 5.2 MB/s | 5.6 MB 00:01 2023-09-30T00:41:28.669 INFO:teuthology.orchestra.run.smithi071.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 503 kB/s | 306 kB 00:00 2023-09-30T00:41:28.814 INFO:teuthology.orchestra.run.smithi071.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 408 kB/s | 59 kB 00:00 2023-09-30T00:41:28.932 INFO:teuthology.orchestra.run.smithi071.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 637 kB/s | 75 kB 00:00 2023-09-30T00:41:28.988 INFO:teuthology.orchestra.run.smithi017.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 139 kB/s | 59 kB 00:00 2023-09-30T00:41:28.992 INFO:teuthology.orchestra.run.smithi017.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 594 kB/s | 306 kB 00:00 2023-09-30T00:41:29.163 INFO:teuthology.orchestra.run.smithi017.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 429 kB/s | 75 kB 00:00 2023-09-30T00:41:29.201 INFO:teuthology.orchestra.run.smithi017.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.1 MB/s | 44 kB 00:00 2023-09-30T00:41:29.217 INFO:teuthology.orchestra.run.smithi017.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 16 MB/s | 259 kB 00:00 2023-09-30T00:41:29.235 INFO:teuthology.orchestra.run.smithi017.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 20 MB/s | 353 kB 00:00 2023-09-30T00:41:29.248 INFO:teuthology.orchestra.run.smithi017.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.5 MB/s | 47 kB 00:00 2023-09-30T00:41:29.371 INFO:teuthology.orchestra.run.smithi017.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 4.6 MB/s | 1.7 MB 00:00 2023-09-30T00:41:29.372 INFO:teuthology.orchestra.run.smithi071.stdout:(15/23): thrift-0.13.0-2.el8.x86_64.rpm 3.9 MB/s | 1.7 MB 00:00 2023-09-30T00:41:29.522 INFO:teuthology.orchestra.run.smithi071.stdout:(16/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 2.6 MB/s | 17 MB 00:06 2023-09-30T00:41:29.524 INFO:teuthology.orchestra.run.smithi071.stdout:(17/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 308 kB/s | 47 kB 00:00 2023-09-30T00:41:29.545 INFO:teuthology.orchestra.run.smithi071.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2023-09-30T00:41:29.556 INFO:teuthology.orchestra.run.smithi071.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2023-09-30T00:41:29.564 INFO:teuthology.orchestra.run.smithi071.stdout:(20/23): librdkafka-0.11.4-1.el8.x86_64.rpm 18 MB/s | 353 kB 00:00 2023-09-30T00:41:29.903 INFO:teuthology.orchestra.run.smithi017.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 6.7 MB/s | 3.6 MB 00:00 2023-09-30T00:41:30.395 INFO:teuthology.orchestra.run.smithi017.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.0 MB/s | 3.5 MB 00:01 2023-09-30T00:41:30.589 INFO:teuthology.orchestra.run.smithi071.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 3.5 MB/s | 3.6 MB 00:01 2023-09-30T00:41:30.609 INFO:teuthology.orchestra.run.smithi071.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.3 MB/s | 3.5 MB 00:01 2023-09-30T00:41:30.812 INFO:teuthology.orchestra.run.smithi017.stdout:(23/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 3.3 MB/s | 23 MB 00:07 2023-09-30T00:41:30.815 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:30.815 INFO:teuthology.orchestra.run.smithi017.stdout:Total 8.8 MB/s | 63 MB 00:07 2023-09-30T00:41:30.820 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:41:30.869 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:41:30.869 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:41:31.644 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:41:31.649 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:41:31.849 INFO:teuthology.orchestra.run.smithi071.stdout:(23/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 2.6 MB/s | 23 MB 00:08 2023-09-30T00:41:31.852 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:31.852 INFO:teuthology.orchestra.run.smithi071.stdout:Total 7.1 MB/s | 63 MB 00:08 2023-09-30T00:41:31.857 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:41:31.901 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:41:31.901 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:41:32.611 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:41:32.646 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:41:32.824 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:41:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:41:33.466 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:41:33.717 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:41:33.876 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:41:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:41:33.963 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:41:34.191 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:41:34.285 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:41:34.432 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T00:41:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:41:34.623 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:41:34.660 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:41:34.800 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:41:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T00:41:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:41:35.243 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:41:35.309 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:41:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:41:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-09-30T00:41:35.646 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:41:35.776 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:41:35.843 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:41:35.944 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:41:36.018 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:41:36.168 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-30T00:41:36.225 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-30T00:41:36.531 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-09-30T00:41:36.702 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T00:41:36.851 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:41:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:41:37.071 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:41:37.171 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T00:41:37.246 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:41:37.342 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:41:37.412 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:41:37.545 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:41:37.605 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:41:37.704 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:41:37.858 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:41:37.894 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:41:38.017 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:41:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:41:38.137 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:41:38.270 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:41:38.276 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:41:38.334 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:41:38.447 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T00:41:38.503 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:41:38.678 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T00:41:38.715 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:41:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:41:39.557 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:41:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:41:44.336 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:41:44.728 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:41:45.134 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:41:45.358 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:41:45.776 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:41:45.951 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:41:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:41:46.159 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:41:46.251 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:41:46.291 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:41:46.327 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:41:46.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:42:04.180 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:42:04.180 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2023-09-30T00:42:04.180 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2023-09-30T00:42:04.180 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /run 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2023-09-30T00:42:04.181 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /sys 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /proc 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /dev 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /run 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /mnt 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /var/tmp 2023-09-30T00:42:04.728 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /home 2023-09-30T00:42:04.729 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /tmp 2023-09-30T00:42:04.729 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /dev 2023-09-30T00:42:04.729 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:05.871 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:42:05.947 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:42:05.947 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:42:06.172 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:42:06.172 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:42:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:42:06.637 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:42:06.637 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:42:06.825 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:42:06.825 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:42:06.951 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:42:06.951 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:42:06.952 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T00:42:06.953 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:42:06.954 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:42:06.955 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:42:07.621 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-30T00:42:07.622 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-09-30T00:42:07.623 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-09-30T00:42:07.623 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:42:07.623 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:42:07.623 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:42:08.289 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:42:08.289 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:54 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:42:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:42:09.185 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:56 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:42:10.351 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:42:10.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:10.462 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2023-09-30T00:42:10.462 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.462 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.462 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:42:10.462 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.463 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:10.464 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:42:10.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2023-09-30T00:42:10.969 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:42:11.190 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:42:11.308 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:11.308 INFO:teuthology.orchestra.run.smithi017.stdout:Upgraded: 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:42:11.309 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.310 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:11.311 INFO:teuthology.orchestra.run.smithi017.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:42:11.311 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:11.311 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:42:11.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-test 2023-09-30T00:42:11.719 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:57 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:42:11.835 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:42:12.713 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:42:13.575 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:42:13.576 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-09-30T00:42:13.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:13.578 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 53 M 2023-09-30T00:42:13.578 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 233 M 2023-09-30T00:42:13.578 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:42:13.591 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 5.1 MB/s | 70 kB 00:00 2023-09-30T00:42:13.627 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.4 MB/s | 161 kB 00:00 2023-09-30T00:42:13.646 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 9.7 MB/s | 187 kB 00:00 2023-09-30T00:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-09-30T00:42:14.087 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 344 kB/s | 175 kB 00:00 2023-09-30T00:42:14.636 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================ 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================ 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T00:42:14.637 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================================ 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout:Install 6 Packages 2023-09-30T00:42:14.638 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:14.639 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 53 M 2023-09-30T00:42:14.639 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 233 M 2023-09-30T00:42:14.639 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:42:14.643 INFO:teuthology.orchestra.run.smithi017.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 20 MB/s | 70 kB 00:00 2023-09-30T00:42:14.682 INFO:teuthology.orchestra.run.smithi017.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.0 MB/s | 161 kB 00:00 2023-09-30T00:42:14.697 INFO:teuthology.orchestra.run.smithi017.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 13 MB/s | 187 kB 00:00 2023-09-30T00:42:14.712 INFO:teuthology.orchestra.run.smithi017.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 20 MB/s | 302 kB 00:00 2023-09-30T00:42:15.141 INFO:teuthology.orchestra.run.smithi017.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 348 kB/s | 175 kB 00:00 2023-09-30T00:42:27.057 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 3.8 MB/s | 52 MB 00:13 2023-09-30T00:42:27.060 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:27.060 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.9 MB/s | 53 MB 00:13 2023-09-30T00:42:27.064 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:42:27.086 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:42:27.086 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:42:27.425 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:42:27.429 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:42:27.929 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:42:28.118 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T00:42:28.331 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:42:28.417 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:42:28.418 INFO:teuthology.orchestra.run.smithi017.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 3.7 MB/s | 52 MB 00:13 2023-09-30T00:42:28.421 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:28.422 INFO:teuthology.orchestra.run.smithi017.stdout:Total 3.8 MB/s | 53 MB 00:13 2023-09-30T00:42:28.426 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:42:28.452 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:42:28.452 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:42:28.572 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T00:42:28.756 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T00:42:28.822 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:42:28.827 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:42:28.917 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:42:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:42:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T00:42:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:42:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:42:29.966 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T00:42:30.161 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T00:42:30.310 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:42:33.884 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:42:34.137 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:42:34.405 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:42:34.405 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:42:34.405 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:42:34.405 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T00:42:34.405 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T00:42:34.405 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T00:42:35.388 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T00:42:35.388 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:21 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:42:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:42:35.934 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:42:36.212 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:42:36.213 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:42:36.213 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:42:36.213 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T00:42:36.213 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T00:42:36.213 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T00:42:37.276 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.5-12.el8.x86_64 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:42:37.374 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:37.375 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:42:37.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2023-09-30T00:42:37.507 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T00:42:37.507 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:24 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:42:37.835 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:42:38.571 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:24 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:42:39.590 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:42:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:39.697 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:42:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: jq-1.5-12.el8.x86_64 2023-09-30T00:42:39.697 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:42:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T00:42:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:42:39.698 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:39.698 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:42:40.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph 2023-09-30T00:42:40.392 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:42:40.395 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================================= 2023-09-30T00:42:40.395 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:42:40.395 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================================= 2023-09-30T00:42:40.395 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:42:40.396 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T00:42:40.397 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:42:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================================= 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout:Install 28 Packages 2023-09-30T00:42:40.399 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:40.400 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 27 M 2023-09-30T00:42:40.400 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 106 M 2023-09-30T00:42:40.400 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:42:40.700 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:42:40.828 INFO:teuthology.orchestra.run.smithi071.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 72 kB/s | 6.4 kB 00:00 2023-09-30T00:42:41.269 INFO:teuthology.orchestra.run.smithi071.stdout:(2/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 4.3 MB/s | 2.3 MB 00:00 2023-09-30T00:42:41.527 INFO:teuthology.orchestra.run.smithi071.stdout:(3/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 5.6 MB/s | 3.9 MB 00:00 2023-09-30T00:42:41.568 INFO:teuthology.orchestra.run.smithi071.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 5.8 MB/s | 238 kB 00:00 2023-09-30T00:42:41.685 INFO:teuthology.orchestra.run.smithi071.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 373 kB/s | 44 kB 00:00 2023-09-30T00:42:41.751 INFO:teuthology.orchestra.run.smithi071.stdout:(6/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.8 MB/s | 185 kB 00:00 2023-09-30T00:42:41.772 INFO:teuthology.orchestra.run.smithi071.stdout:(7/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.5 MB/s | 1.6 MB 00:01 2023-09-30T00:42:41.825 INFO:teuthology.orchestra.run.smithi071.stdout:(8/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.3 MB/s | 173 kB 00:00 2023-09-30T00:42:41.842 INFO:teuthology.orchestra.run.smithi071.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 653 kB/s | 11 kB 00:00 2023-09-30T00:42:41.846 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:28 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:42:41.859 INFO:teuthology.orchestra.run.smithi071.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 1.0 MB/s | 18 kB 00:00 2023-09-30T00:42:41.877 INFO:teuthology.orchestra.run.smithi071.stdout:(11/28): python3-logutils-0.3.5-11.el8.noarch.r 2.7 MB/s | 49 kB 00:00 2023-09-30T00:42:41.895 INFO:teuthology.orchestra.run.smithi071.stdout:(12/28): python3-more-itertools-7.2.0-3.el8.noa 3.3 MB/s | 59 kB 00:00 2023-09-30T00:42:41.966 INFO:teuthology.orchestra.run.smithi071.stdout:(13/28): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2023-09-30T00:42:41.968 INFO:teuthology.orchestra.run.smithi071.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 3.8 MB/s | 283 kB 00:00 2023-09-30T00:42:41.984 INFO:teuthology.orchestra.run.smithi071.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 902 kB/s | 16 kB 00:00 2023-09-30T00:42:41.985 INFO:teuthology.orchestra.run.smithi071.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 1.0 MB/s | 19 kB 00:00 2023-09-30T00:42:42.002 INFO:teuthology.orchestra.run.smithi071.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 1.3 MB/s | 24 kB 00:00 2023-09-30T00:42:42.003 INFO:teuthology.orchestra.run.smithi071.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 1.7 MB/s | 29 kB 00:00 2023-09-30T00:42:42.021 INFO:teuthology.orchestra.run.smithi071.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2023-09-30T00:42:42.040 INFO:teuthology.orchestra.run.smithi071.stdout:(20/28): python3-waitress-1.4.3-1.el8.noarch.rp 6.7 MB/s | 255 kB 00:00 2023-09-30T00:42:42.042 INFO:teuthology.orchestra.run.smithi071.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 12 MB/s | 251 kB 00:00 2023-09-30T00:42:42.058 INFO:teuthology.orchestra.run.smithi071.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 4.7 MB/s | 85 kB 00:00 2023-09-30T00:42:42.060 INFO:teuthology.orchestra.run.smithi071.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 1.3 MB/s | 23 kB 00:00 2023-09-30T00:42:42.193 INFO:teuthology.orchestra.run.smithi071.stdout:(24/28): python3-markupsafe-0.23-19.el8.x86_64. 291 kB/s | 39 kB 00:00 2023-09-30T00:42:42.207 INFO:teuthology.orchestra.run.smithi071.stdout:(25/28): python3-werkzeug-0.12.2-4.el8.noarch.r 3.0 MB/s | 457 kB 00:00 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi071.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi071.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 701 kB/s | 54 kB 00:00 2023-09-30T00:42:42.686 INFO:teuthology.orchestra.run.smithi071.stdout:(28/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 12 MB/s | 17 MB 00:01 2023-09-30T00:42:42.691 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:42.691 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 27 MB 00:02 2023-09-30T00:42:42.696 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:42:42.732 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:42:42.732 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:42:43.059 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:42:43.064 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:42:43.729 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:42:43.732 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================================= 2023-09-30T00:42:43.732 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:42:43.732 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================================= 2023-09-30T00:42:43.732 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:42:43.733 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T00:42:43.734 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================================= 2023-09-30T00:42:43.735 INFO:teuthology.orchestra.run.smithi017.stdout:Install 28 Packages 2023-09-30T00:42:43.736 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:42:43.736 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 27 M 2023-09-30T00:42:43.736 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 106 M 2023-09-30T00:42:43.736 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:42:44.077 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:42:44.126 INFO:teuthology.orchestra.run.smithi017.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 26 kB/s | 6.4 kB 00:00 2023-09-30T00:42:44.339 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T00:42:44.558 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T00:42:44.716 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T00:42:44.930 INFO:teuthology.orchestra.run.smithi017.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.5 MB/s | 1.6 MB 00:01 2023-09-30T00:42:44.935 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T00:42:45.173 INFO:teuthology.orchestra.run.smithi017.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 1.8 MB/s | 2.3 MB 00:01 2023-09-30T00:42:45.226 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T00:42:45.291 INFO:teuthology.orchestra.run.smithi017.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.0 MB/s | 238 kB 00:00 2023-09-30T00:42:45.420 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T00:42:45.590 INFO:teuthology.orchestra.run.smithi017.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 146 kB/s | 44 kB 00:00 2023-09-30T00:42:45.650 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T00:42:45.822 INFO:teuthology.orchestra.run.smithi017.stdout:(6/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 799 kB/s | 185 kB 00:00 2023-09-30T00:42:45.853 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T00:42:45.934 INFO:teuthology.orchestra.run.smithi017.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.5 MB/s | 173 kB 00:00 2023-09-30T00:42:46.001 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T00:42:46.091 INFO:teuthology.orchestra.run.smithi017.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 2.4 MB/s | 384 kB 00:00 2023-09-30T00:42:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T00:42:46.163 INFO:teuthology.orchestra.run.smithi017.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 154 kB/s | 11 kB 00:00 2023-09-30T00:42:46.235 INFO:teuthology.orchestra.run.smithi017.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 244 kB/s | 18 kB 00:00 2023-09-30T00:42:46.311 INFO:teuthology.orchestra.run.smithi017.stdout:(11/28): python3-logutils-0.3.5-11.el8.noarch.r 650 kB/s | 49 kB 00:00 2023-09-30T00:42:46.330 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T00:42:46.386 INFO:teuthology.orchestra.run.smithi017.stdout:(12/28): python3-more-itertools-7.2.0-3.el8.noa 781 kB/s | 59 kB 00:00 2023-09-30T00:42:46.480 INFO:teuthology.orchestra.run.smithi017.stdout:(13/28): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-09-30T00:42:46.548 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T00:42:46.553 INFO:teuthology.orchestra.run.smithi017.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 217 kB/s | 16 kB 00:00 2023-09-30T00:42:46.624 INFO:teuthology.orchestra.run.smithi017.stdout:(15/28): python3-simplegeneric-0.8.1-17.el8.noa 260 kB/s | 19 kB 00:00 2023-09-30T00:42:46.692 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T00:42:46.696 INFO:teuthology.orchestra.run.smithi017.stdout:(16/28): python3-singledispatch-3.4.0.3-18.el8. 332 kB/s | 24 kB 00:00 2023-09-30T00:42:46.769 INFO:teuthology.orchestra.run.smithi017.stdout:(17/28): python3-tempora-1.14.1-5.el8.noarch.rp 401 kB/s | 29 kB 00:00 2023-09-30T00:42:46.843 INFO:teuthology.orchestra.run.smithi017.stdout:(18/28): python3-trustme-0.6.0-4.el8.noarch.rpm 377 kB/s | 27 kB 00:00 2023-09-30T00:42:46.914 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T00:42:46.934 INFO:teuthology.orchestra.run.smithi017.stdout:(19/28): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-09-30T00:42:47.025 INFO:teuthology.orchestra.run.smithi017.stdout:(20/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2023-09-30T00:42:47.103 INFO:teuthology.orchestra.run.smithi017.stdout:(21/28): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-09-30T00:42:47.107 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T00:42:47.175 INFO:teuthology.orchestra.run.smithi017.stdout:(22/28): python3-zc-lockfile-2.0-2.el8.noarch.r 319 kB/s | 23 kB 00:00 2023-09-30T00:42:47.259 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T00:42:47.307 INFO:teuthology.orchestra.run.smithi017.stdout:(23/28): python3-werkzeug-0.12.2-4.el8.noarch.r 3.4 MB/s | 457 kB 00:00 2023-09-30T00:42:47.375 INFO:teuthology.orchestra.run.smithi017.stdout:(24/28): python3-markupsafe-0.23-19.el8.x86_64. 573 kB/s | 39 kB 00:00 2023-09-30T00:42:47.476 INFO:teuthology.orchestra.run.smithi017.stdout:(25/28): python3-mako-1.0.6-13.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2023-09-30T00:42:47.516 INFO:teuthology.orchestra.run.smithi017.stdout:(26/28): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-09-30T00:42:47.531 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T00:42:47.757 INFO:teuthology.orchestra.run.smithi017.stdout:(27/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 5.9 MB/s | 17 MB 00:02 2023-09-30T00:42:47.759 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T00:42:47.955 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T00:42:48.061 INFO:teuthology.orchestra.run.smithi017.stdout:(28/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 1.0 MB/s | 3.9 MB 00:03 2023-09-30T00:42:48.064 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:48.065 INFO:teuthology.orchestra.run.smithi017.stdout:Total 6.3 MB/s | 27 MB 00:04 2023-09-30T00:42:48.070 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:42:48.113 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:42:48.113 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:42:48.231 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T00:42:48.504 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:42:48.505 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T00:42:48.509 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:42:48.764 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T00:42:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T00:42:49.254 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:42:49.740 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:42:49.956 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T00:42:50.183 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T00:42:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T00:42:50.586 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T00:42:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:42:50.887 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:42:50.919 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T00:42:51.282 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:42:51.293 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T00:42:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:42:51.517 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T00:42:51.613 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:42:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T00:42:51.752 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:42:51.770 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:42:51.868 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:42:51.916 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T00:42:52.056 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T00:42:52.242 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T00:42:52.253 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T00:42:52.254 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T00:42:52.255 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T00:42:52.437 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T00:42:52.583 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T00:42:52.817 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T00:42:53.041 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T00:42:53.216 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T00:42:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T00:42:53.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T00:42:53.680 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:42:53.768 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T00:42:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T00:42:54.304 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T00:42:54.601 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T00:42:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T00:42:55.223 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T00:42:55.398 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:42:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:42:57.140 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:42:57.558 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:42:57.563 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:57.657 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T00:42:57.658 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:42:57.659 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:42:57.660 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:42:57.660 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:42:57.660 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:42:57.660 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:42:57.733 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:42:57.797 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2023-09-30T00:42:57.975 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:42:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:42:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:42:58.159 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:42:58.241 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T00:42:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T00:42:58.658 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T00:42:58.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T00:42:59.039 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T00:43:00.286 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:01.100 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:01.580 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:43:01.581 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:43:01.582 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:43:01.679 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2023-09-30T00:43:02.040 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:02.681 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:43:02.791 INFO:teuthology.orchestra.run.smithi017.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:43:02.792 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:02.793 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:43:02.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-base 2023-09-30T00:43:02.947 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:03.329 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:04.756 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:43:04.756 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 80 k 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 363 k 2023-09-30T00:43:04.757 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:43:05.122 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 219 kB/s | 80 kB 00:00 2023-09-30T00:43:05.123 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:05.123 INFO:teuthology.orchestra.run.smithi071.stdout:Total 219 kB/s | 80 kB 00:00 2023-09-30T00:43:05.124 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:43:05.131 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:43:05.132 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:43:05.133 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:43:05.138 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:43:05.185 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:43:05.540 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:52 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:07.934 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:08.445 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:43:08.446 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:43:08.446 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:43:08.536 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install cephadm 2023-09-30T00:43:08.918 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:09.051 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:09.196 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:10.363 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:10.363 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:56 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:10.600 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:12.535 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:43:12.536 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:43:12.536 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:43:12.536 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:43:12.537 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:12.538 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 80 k 2023-09-30T00:43:12.538 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 363 k 2023-09-30T00:43:12.538 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:43:12.761 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 357 kB/s | 80 kB 00:00 2023-09-30T00:43:12.761 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:12.762 INFO:teuthology.orchestra.run.smithi017.stdout:Total 355 kB/s | 80 kB 00:00 2023-09-30T00:43:12.762 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:43:12.771 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:43:12.771 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:43:12.773 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:43:12.778 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:43:12.825 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:43:15.905 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:43:15.963 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:15.999 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:15.999 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:43:15.999 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:16.000 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:16.000 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:43:16.101 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:16.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:43:16.313 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:16.498 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:17.472 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:43:17.473 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:04 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:18.372 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:04 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:20.139 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:43:20.178 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:43:20.179 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:43:20.180 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:20.180 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 161 k 2023-09-30T00:43:20.180 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 490 k 2023-09-30T00:43:20.180 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:43:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:20.245 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:43:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:20.245 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:43:20.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:43:20.569 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 415 kB/s | 161 kB 00:00 2023-09-30T00:43:20.570 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:20.570 INFO:teuthology.orchestra.run.smithi071.stdout:Total 413 kB/s | 161 kB 00:00 2023-09-30T00:43:20.571 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:43:20.583 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:43:20.583 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:43:20.635 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:43:20.640 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:43:20.753 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:43:20.776 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:20.904 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:43:21.319 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:43:22.304 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:43:22.305 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:08 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:22.639 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:09 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:24.568 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:43:24.569 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:43:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2023-09-30T00:43:24.569 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 161 k 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 490 k 2023-09-30T00:43:24.570 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:43:24.828 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 625 kB/s | 161 kB 00:00 2023-09-30T00:43:24.829 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:24.829 INFO:teuthology.orchestra.run.smithi017.stdout:Total 622 kB/s | 161 kB 00:00 2023-09-30T00:43:24.830 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:43:24.843 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:43:24.843 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:43:24.901 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:43:24.906 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:43:25.022 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:43:25.188 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:43:25.607 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:43:26.682 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:43:26.682 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:13 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:26.723 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:43:26.818 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:43:26.819 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:26.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:43:26.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2023-09-30T00:43:27.317 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:29.198 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:15 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:29.708 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:43:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:29.814 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:43:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:29.815 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:43:29.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr 2023-09-30T00:43:30.322 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:30.535 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:31.016 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:43:31.017 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:43:31.017 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:43:31.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:43:31.411 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:18 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:31.471 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:32.895 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:32.920 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:18 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:33.417 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:43:33.419 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:43:33.419 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:43:33.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:43:33.901 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:34.718 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout:=================================================================================================== 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout:=================================================================================================== 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:43:34.719 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout:=================================================================================================== 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:34.720 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2023-09-30T00:43:34.721 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 8.0 M 2023-09-30T00:43:34.721 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:43:34.861 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 131 kB/s | 14 kB 00:00 2023-09-30T00:43:34.869 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 194 kB/s | 23 kB 00:00 2023-09-30T00:43:34.958 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 348 kB/s | 34 kB 00:00 2023-09-30T00:43:35.030 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-09-30T00:43:35.154 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 4.2 MB/s | 1.7 MB 00:00 2023-09-30T00:43:35.287 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 129 kB/s | 43 kB 00:00 2023-09-30T00:43:35.291 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:35.291 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2023-09-30T00:43:35.293 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:43:35.300 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:43:35.301 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:43:35.383 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:43:35.387 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:43:35.673 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:43:35.862 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T00:43:35.935 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:22 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:36.067 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T00:43:36.228 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T00:43:36.373 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:43:36.753 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:43:36.976 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:43:37.233 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:43:37.234 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:43:37.234 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:43:37.234 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:43:37.234 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T00:43:37.234 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T00:43:37.832 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:43:37.833 INFO:teuthology.orchestra.run.smithi017.stdout:=================================================================================================== 2023-09-30T00:43:37.833 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:43:37.833 INFO:teuthology.orchestra.run.smithi017.stdout:=================================================================================================== 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:37.834 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:43:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:=================================================================================================== 2023-09-30T00:43:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:Install 6 Packages 2023-09-30T00:43:37.835 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 2.0 M 2023-09-30T00:43:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 8.0 M 2023-09-30T00:43:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:43:38.126 INFO:teuthology.orchestra.run.smithi017.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 126 kB/s | 14 kB 00:00 2023-09-30T00:43:38.197 INFO:teuthology.orchestra.run.smithi017.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 124 kB/s | 23 kB 00:00 2023-09-30T00:43:38.215 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T00:43:38.215 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:24 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:38.249 INFO:teuthology.orchestra.run.smithi017.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 7.1 MB/s | 1.7 MB 00:00 2023-09-30T00:43:38.350 INFO:teuthology.orchestra.run.smithi017.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 150 kB/s | 34 kB 00:00 2023-09-30T00:43:38.385 INFO:teuthology.orchestra.run.smithi017.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 313 kB/s | 43 kB 00:00 2023-09-30T00:43:38.554 INFO:teuthology.orchestra.run.smithi017.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 551 kB/s | 196 kB 00:00 2023-09-30T00:43:38.557 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:38.557 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2023-09-30T00:43:38.559 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:43:38.568 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:43:38.568 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:43:38.662 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:43:38.667 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:43:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:43:39.294 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T00:43:39.523 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T00:43:39.722 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T00:43:39.835 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:43:40.222 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:43:40.501 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:43:40.763 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:43:40.763 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:43:40.763 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:43:40.763 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:43:40.764 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T00:43:40.764 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T00:43:41.776 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:43:41.866 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T00:43:41.867 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:28 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:41.873 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:41.873 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:41.874 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:43:41.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:43:42.350 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:44.231 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:30 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:46.052 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:43:46.053 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================================================== 2023-09-30T00:43:46.053 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================================================== 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T00:43:46.054 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================================================== 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout:Install 8 Packages 2023-09-30T00:43:46.055 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:43:46.056 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 36 M 2023-09-30T00:43:46.056 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 199 M 2023-09-30T00:43:46.056 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:43:46.083 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:43:46.158 INFO:teuthology.orchestra.run.smithi071.stdout:(1/8): libgfortran-8.4.1-1.el8.x86_64.rpm 6.1 MB/s | 639 kB 00:00 2023-09-30T00:43:46.189 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:46.189 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:43:46.189 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:46.189 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:46.189 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:43:46.190 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:43:46.190 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:43:46.190 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:43:46.190 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:46.190 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:43:46.193 INFO:teuthology.orchestra.run.smithi071.stdout:(2/8): libquadmath-8.4.1-1.el8.x86_64.rpm 1.2 MB/s | 169 kB 00:00 2023-09-30T00:43:46.299 INFO:teuthology.orchestra.run.smithi071.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 2.1 MB/s | 225 kB 00:00 2023-09-30T00:43:46.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:43:46.454 INFO:teuthology.orchestra.run.smithi071.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2023-09-30T00:43:46.562 INFO:teuthology.orchestra.run.smithi071.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 36 MB/s | 15 MB 00:00 2023-09-30T00:43:46.615 INFO:teuthology.orchestra.run.smithi071.stdout:(6/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2023-09-30T00:43:46.658 INFO:teuthology.orchestra.run.smithi071.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 48 MB/s | 4.6 MB 00:00 2023-09-30T00:43:46.696 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:43:46.808 INFO:teuthology.orchestra.run.smithi071.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 9.8 MB/s | 7.4 MB 00:00 2023-09-30T00:43:46.809 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:46.810 INFO:teuthology.orchestra.run.smithi071.stdout:Total 48 MB/s | 36 MB 00:00 2023-09-30T00:43:46.819 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:43:46.838 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:43:46.838 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:43:47.259 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:43:47.264 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:43:47.847 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:43:48.031 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:43:48.175 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:43:48.247 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:43:48.314 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:43:48.860 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:43:49.056 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:43:49.585 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:43:49.839 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:43:50.483 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================================================== 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================================================== 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T00:43:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================================================== 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout:Install 8 Packages 2023-09-30T00:43:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:43:50.487 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 36 M 2023-09-30T00:43:50.487 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 199 M 2023-09-30T00:43:50.487 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:43:50.616 INFO:teuthology.orchestra.run.smithi017.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 1.7 MB/s | 225 kB 00:00 2023-09-30T00:43:50.880 INFO:teuthology.orchestra.run.smithi017.stdout:(2/8): python3-scipy-1.0.0-20.module+el8.1.0+33 37 MB/s | 15 MB 00:00 2023-09-30T00:43:50.895 INFO:teuthology.orchestra.run.smithi017.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2023-09-30T00:43:50.940 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:43:51.017 INFO:teuthology.orchestra.run.smithi017.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2023-09-30T00:43:51.083 INFO:teuthology.orchestra.run.smithi017.stdout:(5/8): openblas-0.3.12-1.el8.x86_64.rpm 24 MB/s | 4.6 MB 00:00 2023-09-30T00:43:51.116 INFO:teuthology.orchestra.run.smithi017.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 1.7 MB/s | 169 kB 00:00 2023-09-30T00:43:51.147 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:43:51.217 INFO:teuthology.orchestra.run.smithi017.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 4.7 MB/s | 639 kB 00:00 2023-09-30T00:43:51.478 INFO:teuthology.orchestra.run.smithi017.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 7.4 MB/s | 7.4 MB 00:00 2023-09-30T00:43:51.480 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:51.480 INFO:teuthology.orchestra.run.smithi017.stdout:Total 36 MB/s | 36 MB 00:00 2023-09-30T00:43:51.489 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:43:51.981 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:43:51.985 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:43:52.676 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:43:52.867 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:43:53.018 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:43:53.142 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:43:53.197 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:43:53.767 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:43:53.950 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:43:54.445 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T00:43:54.514 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:43:54.629 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:43:54.750 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:43:54.941 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:43:54.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:43:54.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:43:54.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-09-30T00:43:54.942 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-09-30T00:43:54.942 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:43:54.942 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:43:54.942 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-09-30T00:43:55.540 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:43:55.874 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:43:55.937 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-09-30T00:43:55.937 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:43:56.092 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:43:58.298 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:43:59.614 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T00:43:59.816 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-09-30T00:44:00.115 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-09-30T00:44:01.177 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:44:01.251 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-09-30T00:44:01.251 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:01.277 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:01.277 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:44:01.277 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:01.277 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:01.278 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:01.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:44:01.776 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:03.632 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:49 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:05.459 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================================== 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================================== 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:44:05.461 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:44:05.462 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================================== 2023-09-30T00:44:05.463 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2023-09-30T00:44:05.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:05.463 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.9 M 2023-09-30T00:44:05.463 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 25 M 2023-09-30T00:44:05.463 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:44:05.576 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 288 kB/s | 16 kB 00:00 2023-09-30T00:44:05.583 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 518 kB/s | 33 kB 00:00 2023-09-30T00:44:05.600 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 3.3 MB/s | 81 kB 00:00 2023-09-30T00:44:05.619 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 3.3 MB/s | 64 kB 00:00 2023-09-30T00:44:05.638 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 3.3 MB/s | 61 kB 00:00 2023-09-30T00:44:05.650 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-kubernetes-11.0.0-6.el8.noarch. 17 MB/s | 1.2 MB 00:00 2023-09-30T00:44:05.678 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.8 MB/s | 155 kB 00:00 2023-09-30T00:44:05.693 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2023-09-30T00:44:05.706 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 967 kB/s | 27 kB 00:00 2023-09-30T00:44:05.724 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.0 MB/s | 126 kB 00:00 2023-09-30T00:44:05.725 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 5.5 MB/s | 110 kB 00:00 2023-09-30T00:44:05.740 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el 214 kB/s | 47 kB 00:00 2023-09-30T00:44:05.753 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 685 kB/s | 20 kB 00:00 2023-09-30T00:44:05.756 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:05.756 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.6 MB/s | 1.9 MB 00:00 2023-09-30T00:44:05.762 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:44:05.776 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:44:05.776 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:44:05.864 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:44:05.869 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:44:05.935 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:44:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T00:44:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T00:44:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T00:44:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:06.046 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:44:06.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:44:06.393 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:44:06.570 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:44:06.867 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T00:44:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T00:44:07.276 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-09-30T00:44:07.503 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-09-30T00:44:07.590 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:07.758 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:44:07.989 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:44:08.243 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T00:44:08.450 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:44:08.683 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T00:44:09.500 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:44:09.502 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================================== 2023-09-30T00:44:09.502 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:44:09.502 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================================== 2023-09-30T00:44:09.502 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:44:09.502 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:44:09.502 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:44:09.503 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout:======================================================================================================== 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout:Install 13 Packages 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:09.504 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 1.9 M 2023-09-30T00:44:09.505 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 25 M 2023-09-30T00:44:09.505 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:44:09.798 INFO:teuthology.orchestra.run.smithi017.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 205 kB/s | 47 kB 00:00 2023-09-30T00:44:09.866 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T00:44:09.886 INFO:teuthology.orchestra.run.smithi017.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 51 kB/s | 16 kB 00:00 2023-09-30T00:44:09.911 INFO:teuthology.orchestra.run.smithi017.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 96 kB/s | 33 kB 00:00 2023-09-30T00:44:10.073 INFO:teuthology.orchestra.run.smithi017.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 397 kB/s | 64 kB 00:00 2023-09-30T00:44:10.178 INFO:teuthology.orchestra.run.smithi017.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 585 kB/s | 61 kB 00:00 2023-09-30T00:44:10.187 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T00:44:10.221 INFO:teuthology.orchestra.run.smithi017.stdout:(6/13): python3-google-auth-1.1.1-10.el8.noarch 192 kB/s | 81 kB 00:00 2023-09-30T00:44:10.328 INFO:teuthology.orchestra.run.smithi017.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 177 kB/s | 27 kB 00:00 2023-09-30T00:44:10.385 INFO:teuthology.orchestra.run.smithi017.stdout:(8/13): python3-kubernetes-11.0.0-6.el8.noarch. 2.3 MB/s | 1.2 MB 00:00 2023-09-30T00:44:10.413 INFO:teuthology.orchestra.run.smithi017.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 662 kB/s | 126 kB 00:00 2023-09-30T00:44:10.460 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:44:10.474 INFO:teuthology.orchestra.run.smithi017.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 752 kB/s | 110 kB 00:00 2023-09-30T00:44:10.497 INFO:teuthology.orchestra.run.smithi017.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 179 kB/s | 20 kB 00:00 2023-09-30T00:44:10.521 INFO:teuthology.orchestra.run.smithi017.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.4 MB/s | 155 kB 00:00 2023-09-30T00:44:10.665 INFO:teuthology.orchestra.run.smithi017.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 225 kB/s | 43 kB 00:00 2023-09-30T00:44:10.669 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:10.669 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-09-30T00:44:10.675 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:44:10.692 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:44:10.692 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T00:44:10.717 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T00:44:10.718 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:44:10.718 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:44:10.718 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-09-30T00:44:10.718 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-09-30T00:44:10.718 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-09-30T00:44:10.781 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:44:10.786 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:44:11.313 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:44:11.574 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:44:11.746 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-09-30T00:44:11.747 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:57 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T00:44:12.064 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T00:44:12.332 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-09-30T00:44:12.572 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-09-30T00:44:12.807 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-09-30T00:44:12.992 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-09-30T00:44:13.225 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T00:44:13.411 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:44:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T00:44:14.915 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T00:44:15.210 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T00:44:15.349 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:44:15.447 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:15.447 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:44:15.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:44:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:44:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:44:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:44:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:15.449 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:15.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-09-30T00:44:15.948 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:16.802 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-09-30T00:44:16.802 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:03 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:16.863 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:02 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:18.654 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:44:18.654 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================================== 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================================== 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:44:18.655 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================================== 2023-09-30T00:44:18.656 INFO:teuthology.orchestra.run.smithi071.stdout:Install 5 Packages 2023-09-30T00:44:18.656 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:18.656 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.0 M 2023-09-30T00:44:18.656 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2023-09-30T00:44:18.656 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:44:18.977 INFO:teuthology.orchestra.run.smithi071.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 801 kB/s | 121 kB 00:00 2023-09-30T00:44:19.072 INFO:teuthology.orchestra.run.smithi071.stdout:(2/5): python3-babel-2.5.1-5.el8.noarch.rpm 50 MB/s | 4.8 MB 00:00 2023-09-30T00:44:19.100 INFO:teuthology.orchestra.run.smithi071.stdout:(3/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 19 MB/s | 538 kB 00:00 2023-09-30T00:44:19.171 INFO:teuthology.orchestra.run.smithi071.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 175 kB/s | 60 kB 00:00 2023-09-30T00:44:19.359 INFO:teuthology.orchestra.run.smithi071.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 893 kB/s | 476 kB 00:00 2023-09-30T00:44:19.360 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:19.360 INFO:teuthology.orchestra.run.smithi071.stdout:Total 8.4 MB/s | 6.0 MB 00:00 2023-09-30T00:44:19.363 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:44:19.373 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:44:19.373 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:44:19.462 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:44:19.464 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:44:19.469 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:44:19.568 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:19.568 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:44:19.568 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:19.568 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:44:19.569 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:44:19.570 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:44:19.570 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:44:19.570 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:44:19.570 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:44:19.570 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:19.570 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:44:19.706 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:44:20.100 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:20.495 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:44:21.712 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T00:44:21.865 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:08 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:21.969 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T00:44:22.289 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:44:22.528 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:44:22.748 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:44:23.017 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:44:23.017 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:44:23.017 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:44:23.017 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:44:23.018 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T00:44:23.735 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:44:23.736 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================================== 2023-09-30T00:44:23.736 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================================== 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:44:23.737 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================================== 2023-09-30T00:44:23.738 INFO:teuthology.orchestra.run.smithi017.stdout:Install 5 Packages 2023-09-30T00:44:23.738 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:23.738 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 6.0 M 2023-09-30T00:44:23.738 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 26 M 2023-09-30T00:44:23.738 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:44:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T00:44:23.923 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:09 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:24.058 INFO:teuthology.orchestra.run.smithi017.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 469 kB/s | 121 kB 00:00 2023-09-30T00:44:24.203 INFO:teuthology.orchestra.run.smithi017.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 149 kB/s | 60 kB 00:00 2023-09-30T00:44:24.353 INFO:teuthology.orchestra.run.smithi017.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2023-09-30T00:44:24.356 INFO:teuthology.orchestra.run.smithi017.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 3.4 MB/s | 538 kB 00:00 2023-09-30T00:44:24.457 INFO:teuthology.orchestra.run.smithi017.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 724 kB/s | 476 kB 00:00 2023-09-30T00:44:24.459 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:24.459 INFO:teuthology.orchestra.run.smithi017.stdout:Total 8.2 MB/s | 6.0 MB 00:00 2023-09-30T00:44:24.463 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:44:24.474 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:44:24.474 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:44:24.576 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:44:24.581 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:44:25.636 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:44:26.951 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T00:44:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T00:44:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:44:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:44:27.958 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:44:27.995 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T00:44:28.057 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:44:28.058 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:28.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2023-09-30T00:44:28.258 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:44:28.259 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:44:28.259 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:44:28.259 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:44:28.259 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T00:44:28.542 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:16 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:15 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:44:31.435 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:44:31.436 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:31.437 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2023-09-30T00:44:31.437 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:31.437 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 957 k 2023-09-30T00:44:31.437 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.0 M 2023-09-30T00:44:31.437 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:44:31.602 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 500 kB/s | 83 kB 00:00 2023-09-30T00:44:31.806 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 874 kB 00:00 2023-09-30T00:44:31.806 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:31.807 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.5 MB/s | 957 kB 00:00 2023-09-30T00:44:31.808 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:44:31.912 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:44:31.916 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:44:32.082 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:44:32.382 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T00:44:32.529 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:44:32.901 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:44:32.904 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:44:32.905 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T00:44:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:44:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:33.011 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:44:33.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-fuse 2023-09-30T00:44:33.513 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:33.897 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T00:44:33.897 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:19 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:35.256 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:22 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:37.163 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:44:37.163 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:37.164 INFO:teuthology.orchestra.run.smithi017.stdout:Install 2 Packages 2023-09-30T00:44:37.165 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:37.165 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 957 k 2023-09-30T00:44:37.165 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 3.0 M 2023-09-30T00:44:37.165 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:44:37.225 INFO:teuthology.orchestra.run.smithi017.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.3 MB/s | 83 kB 00:00 2023-09-30T00:44:37.760 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:44:37.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:37.858 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:44:37.858 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T00:44:37.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:37.858 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:38.336 INFO:teuthology.orchestra.run.smithi017.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 747 kB/s | 874 kB 00:01 2023-09-30T00:44:38.336 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:38.337 INFO:teuthology.orchestra.run.smithi017.stdout:Total 816 kB/s | 957 kB 00:01 2023-09-30T00:44:38.339 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:44:38.362 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:44:38.362 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:44:38.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2023-09-30T00:44:38.453 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:44:38.457 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:44:38.651 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:44:38.741 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:38.904 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T00:44:39.061 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:44:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:44:39.471 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:44:40.301 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:26 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:40.585 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T00:44:40.585 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:27 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:42.083 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:42.084 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:44:42.085 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:42.085 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 267 k 2023-09-30T00:44:42.085 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.3 M 2023-09-30T00:44:42.085 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:44:42.258 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-09-30T00:44:42.259 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:42.259 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-09-30T00:44:42.259 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:44:42.270 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:44:42.270 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:44:42.298 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:44:42.302 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:44:42.481 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:44:42.819 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:44:43.242 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:44:43.663 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:44:43.770 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:43.770 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:44:43.770 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T00:44:43.770 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:43.770 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:44:43.880 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-volume 2023-09-30T00:44:44.279 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:44.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:44:44.334 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:30 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:45.484 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:32 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:46.438 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:44:46.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:46.538 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:44:46.538 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:46.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:46.538 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:46.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2023-09-30T00:44:47.037 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:47.622 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:44:47.623 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:47.624 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 267 k 2023-09-30T00:44:47.624 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 1.3 M 2023-09-30T00:44:47.624 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:44:47.886 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:33 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:47.917 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 910 kB/s | 267 kB 00:00 2023-09-30T00:44:47.917 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:47.918 INFO:teuthology.orchestra.run.smithi017.stdout:Total 905 kB/s | 267 kB 00:00 2023-09-30T00:44:47.918 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:44:47.930 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:44:47.930 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:44:47.956 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:44:47.961 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:44:48.158 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:44:48.819 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:44:49.311 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:44:49.731 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:44:49.732 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-09-30T00:44:49.732 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:49.732 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:44:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:44:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 130 k 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 464 k 2023-09-30T00:44:49.733 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:44:49.990 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 505 kB/s | 130 kB 00:00 2023-09-30T00:44:49.990 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:49.991 INFO:teuthology.orchestra.run.smithi071.stdout:Total 503 kB/s | 130 kB 00:00 2023-09-30T00:44:49.991 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:44:49.998 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:44:49.998 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:44:50.053 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:44:50.058 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:44:50.173 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:44:50.324 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:50.417 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:44:50.417 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:37 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:51.546 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:51.546 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:37 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:44:52.889 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:52.889 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:44:52.889 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:44:52.889 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:52.889 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:44:53.008 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados-devel 2023-09-30T00:44:53.406 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:53.632 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:44:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:53.727 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:44:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:44:53.727 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:53.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2023-09-30T00:44:54.210 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:54.388 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:41 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:55.392 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:41 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:44:56.358 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:44:56.359 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2023-09-30T00:44:56.359 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:56.359 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:44:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 130 k 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 464 k 2023-09-30T00:44:56.360 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:44:56.623 INFO:teuthology.orchestra.run.smithi017.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 495 kB/s | 130 kB 00:00 2023-09-30T00:44:56.623 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:56.624 INFO:teuthology.orchestra.run.smithi017.stdout:Total 492 kB/s | 130 kB 00:00 2023-09-30T00:44:56.624 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:44:56.631 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:44:56.632 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:44:56.692 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:44:56.696 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:44:56.708 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:44:56.826 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:44:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:57.182 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:44:57.183 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:44:57.183 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:44:57.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2023-09-30T00:44:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:57.651 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:44:58.437 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:58.437 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:45 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:44:59.341 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:45 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:01.195 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:01.195 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:01.195 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-09-30T00:45:01.195 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:45:01.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:01.197 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 25 k 2023-09-30T00:45:01.197 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 109 k 2023-09-30T00:45:01.197 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:45:01.308 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 224 kB/s | 25 kB 00:00 2023-09-30T00:45:01.308 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:01.309 INFO:teuthology.orchestra.run.smithi071.stdout:Total 221 kB/s | 25 kB 00:00 2023-09-30T00:45:01.309 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:45:01.312 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:45:01.313 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:45:01.314 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:45:01.319 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:45:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:45:01.547 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:01.786 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:02.613 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:45:02.720 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:02.720 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:45:02.720 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:02.720 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:02.721 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:02.738 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:02.738 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:48 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:02.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs2 2023-09-30T00:45:03.254 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:04.991 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:51 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:06.170 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:45:06.266 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:06.267 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:45:06.267 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:06.267 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:06.267 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:06.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2023-09-30T00:45:06.400 INFO:teuthology.orchestra.run.smithi017.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:06.759 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:06.895 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:06.897 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:06.897 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:06.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs-devel 2023-09-30T00:45:07.389 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:08.449 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:54 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:09.176 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:56 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:09.763 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:10.245 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:10.246 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:45:10.246 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:10.398 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2023-09-30T00:45:10.770 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:11.137 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:11.138 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2023-09-30T00:45:11.138 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 25 k 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 109 k 2023-09-30T00:45:11.139 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:45:11.250 INFO:teuthology.orchestra.run.smithi017.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 224 kB/s | 25 kB 00:00 2023-09-30T00:45:11.250 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:11.251 INFO:teuthology.orchestra.run.smithi017.stdout:Total 221 kB/s | 25 kB 00:00 2023-09-30T00:45:11.251 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:45:11.255 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:45:11.255 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:45:11.257 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:45:11.261 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:45:11.323 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:45:11.448 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:11.705 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:12.435 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:58 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:12.827 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:12.828 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:03:59 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:13.754 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:14.231 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:14.232 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:45:14.232 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:14.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2023-09-30T00:45:14.699 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:16.252 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:45:16.359 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:16.359 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:45:16.359 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:16.359 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:16.359 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:16.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados2 2023-09-30T00:45:16.473 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:02 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:16.862 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:17.795 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:18.263 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:18.264 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:45:18.265 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:18.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2023-09-30T00:45:18.521 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:05 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:18.742 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:19.917 INFO:teuthology.orchestra.run.smithi017.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:20.292 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:06 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:20.411 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:20.413 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:20.413 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:20.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librbd1 2023-09-30T00:45:20.950 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:21.610 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:22.081 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:22.082 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:45:22.083 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:22.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2023-09-30T00:45:22.550 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:22.615 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:09 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:24.023 INFO:teuthology.orchestra.run.smithi017.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:24.102 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:10 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:24.521 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:24.523 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:24.523 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:24.625 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rados 2023-09-30T00:45:25.020 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:25.452 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:25.923 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:25.925 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:45:25.925 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:26.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2023-09-30T00:45:26.398 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:26.546 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:13 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:27.980 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:28.012 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:14 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:28.483 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:28.484 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:28.484 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:28.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rgw 2023-09-30T00:45:28.975 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:29.376 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2023-09-30T00:45:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:29.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2023-09-30T00:45:30.316 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:30.678 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:17 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:31.970 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:17 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:32.063 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:32.565 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:32.566 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:32.566 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:32.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-cephfs 2023-09-30T00:45:33.068 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:33.761 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:33.762 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:33.762 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 90 k 2023-09-30T00:45:33.763 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 264 k 2023-09-30T00:45:33.764 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:45:33.996 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:20 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:34.115 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 257 kB/s | 90 kB 00:00 2023-09-30T00:45:34.115 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:34.116 INFO:teuthology.orchestra.run.smithi071.stdout:Total 256 kB/s | 90 kB 00:00 2023-09-30T00:45:34.116 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:45:34.127 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:45:34.128 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:45:34.183 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:45:34.187 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:45:34.300 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:45:34.490 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:34.756 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:35.579 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:35.695 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:35.695 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:21 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:36.079 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:36.080 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:36.081 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:36.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rbd 2023-09-30T00:45:36.722 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:37.604 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:45:37.699 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:37.699 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:45:37.699 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:37.700 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:37.700 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:37.704 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:24 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:37.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2023-09-30T00:45:38.191 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:39.009 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:25 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:39.162 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:45:39.661 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:39.662 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2023-09-30T00:45:39.662 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:39.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-fuse 2023-09-30T00:45:40.162 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:40.874 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:40.875 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:45:40.876 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:40.876 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:45:40.876 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:40.876 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.5 M 2023-09-30T00:45:40.876 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 16 M 2023-09-30T00:45:40.876 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:45:41.064 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:28 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:41.240 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 9.5 MB/s | 3.5 MB 00:00 2023-09-30T00:45:41.241 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:41.242 INFO:teuthology.orchestra.run.smithi071.stdout:Total 9.4 MB/s | 3.5 MB 00:00 2023-09-30T00:45:41.242 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:45:41.255 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:45:41.255 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:45:41.325 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:45:41.329 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:45:41.782 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:45:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:42.356 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:42.987 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:42.988 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:42.988 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2023-09-30T00:45:42.988 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:42.988 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:45:42.988 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:45:42.988 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:42.989 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:45:42.989 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:42.989 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:45:42.989 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:42.989 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 90 k 2023-09-30T00:45:42.989 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 264 k 2023-09-30T00:45:42.990 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:45:43.212 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 406 kB/s | 90 kB 00:00 2023-09-30T00:45:43.212 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:43.212 INFO:teuthology.orchestra.run.smithi017.stdout:Total 404 kB/s | 90 kB 00:00 2023-09-30T00:45:43.213 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:45:43.226 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:45:43.226 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:45:43.267 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:43.267 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:29 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:43.286 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:45:43.291 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:45:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:45:43.644 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:43.921 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:45.012 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:45.012 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:32 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:45.240 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:45:45.337 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:45.337 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:45:45.337 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:45.337 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:45.337 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:45.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2023-09-30T00:45:45.837 INFO:teuthology.orchestra.run.smithi071.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:46.623 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:32 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:47.010 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:45:47.118 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:47.118 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:45:47.118 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:47.118 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:47.118 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:47.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-mirror 2023-09-30T00:45:47.635 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:48.414 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:48.415 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 173 k 2023-09-30T00:45:48.416 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 552 k 2023-09-30T00:45:48.416 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-09-30T00:45:48.593 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:35 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:48.694 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 621 kB/s | 173 kB 00:00 2023-09-30T00:45:48.695 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:48.695 INFO:teuthology.orchestra.run.smithi071.stdout:Total 618 kB/s | 173 kB 00:00 2023-09-30T00:45:48.696 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-09-30T00:45:48.708 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-09-30T00:45:48.709 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-09-30T00:45:48.765 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-09-30T00:45:48.769 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-09-30T00:45:48.889 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-09-30T00:45:49.053 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:50.244 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:36 ago on Sat 30 Sep 2023 12:41:14 AM UTC. 2023-09-30T00:45:50.526 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:45:50.527 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2023-09-30T00:45:50.527 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:50.527 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:45:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:45:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:50.528 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:45:50.528 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:45:50.528 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:45:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:50.528 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 3.5 M 2023-09-30T00:45:50.528 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 16 M 2023-09-30T00:45:50.529 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:45:52.160 INFO:teuthology.orchestra.run.smithi071.stdout:Installed products updated. 2023-09-30T00:45:52.258 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:52.259 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-09-30T00:45:52.259 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:52.259 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:45:52.259 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-09-30T00:45:52.381 DEBUG:teuthology.parallel:result is None 2023-09-30T00:45:52.706 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 1.6 MB/s | 3.5 MB 00:02 2023-09-30T00:45:52.707 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:52.707 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.6 MB/s | 3.5 MB 00:02 2023-09-30T00:45:52.708 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:45:52.725 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:45:52.725 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:45:52.802 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:45:52.807 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:45:53.310 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:45:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:53.952 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:55.023 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:55.023 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:42 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:45:57.011 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:45:57.118 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:57.118 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:45:57.118 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:57.119 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:45:57.119 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:45:57.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-nbd 2023-09-30T00:45:57.637 INFO:teuthology.orchestra.run.smithi017.stdout:Updating Subscription Management repositories. 2023-09-30T00:45:58.564 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:45 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:46:00.454 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2023-09-30T00:46:00.455 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:00.456 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 173 k 2023-09-30T00:46:00.456 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 552 k 2023-09-30T00:46:00.456 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2023-09-30T00:46:00.714 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 671 kB/s | 173 kB 00:00 2023-09-30T00:46:00.714 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:46:00.714 INFO:teuthology.orchestra.run.smithi017.stdout:Total 667 kB/s | 173 kB 00:00 2023-09-30T00:46:00.715 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2023-09-30T00:46:00.729 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2023-09-30T00:46:00.729 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2023-09-30T00:46:00.790 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2023-09-30T00:46:00.794 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2023-09-30T00:46:00.923 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2023-09-30T00:46:01.109 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:46:01.392 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:46:02.473 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:46:02.473 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:04:49 ago on Sat 30 Sep 2023 12:41:13 AM UTC. 2023-09-30T00:46:04.508 INFO:teuthology.orchestra.run.smithi017.stdout:Installed products updated. 2023-09-30T00:46:04.611 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:04.611 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2023-09-30T00:46:04.612 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:46:04.612 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:04.612 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2023-09-30T00:46:04.730 DEBUG:teuthology.parallel:result is None 2023-09-30T00:46:04.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:46:04.959 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:46:04.980 INFO:teuthology.orchestra.run.smithi017.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:46:04.980 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:46:04.980 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:46:04.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:46:05.200 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:46:05.221 INFO:teuthology.orchestra.run.smithi071.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:46:05.222 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:46:05.222 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:46:05.222 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:46:05.222 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:05.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:46:05.249 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:05.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:46:05.288 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:46:05.289 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:05.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:46:05.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:46:05.379 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:05.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:46:05.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:46:05.468 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:46:05.468 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:05.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:46:05.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:46:05.557 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:05.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:46:05.582 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:46:05.646 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:46:05.647 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:05.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:46:05.672 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:46:05.737 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:05.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:46:05.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:46:05.824 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:46:05.884 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:46:05.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2023-09-30T00:46:05.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2023-09-30T00:46:05.910 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:46:05.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:46:05.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:46:05.975 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:46:05.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:46:06.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:46:06.051 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:46:06.051 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:46:06.052 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406764', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZlaQEP+7adkueMb/+ERFQy3Dwhho1Wb12ZhXccY71Dnj1nVL4Fj5qVr8Ipasufq3xp01VCo5NHXPqmMBXL7Sc=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJJ2MZTCXYCd/ZJ7xqPR+owGRouh0wJqcJ14wyaj2b535kQMDTxSikwVVcIMqPOdR3lKn5wcwb3bg42m1dINiU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 3600', 'sudo ceph osd pool set cache target_max_objects 250', 'sudo ceph osd pool set cache min_read_recency_for_promote 2']}}, {'rados': {'balance_reads': True, 'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:46:06.052 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:46:06.081 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:46:06.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:46:06.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:46:06.160 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:06.160 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:46:06.214 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:46:06.214 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2023-09-30T00:46:06.269 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 215719 Links: 1 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-09-30 00:45:53.711032043 +0000 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-09-30 00:40:28.032255732 +0000 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-09-30 00:40:28.032255732 +0000 2023-09-30T00:46:06.270 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-09-30T00:46:06.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:46:06.333 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-09-30T00:46:06.333 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-09-30T00:46:06.334 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000116259 s, 4.4 MB/s 2023-09-30T00:46:06.334 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:46:06.391 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2023-09-30T00:46:06.446 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:46:06.446 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:06.446 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 218901 Links: 1 2023-09-30T00:46:06.446 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:06.447 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:06.447 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-09-30 00:45:53.711032043 +0000 2023-09-30T00:46:06.447 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-09-30 00:40:28.516251133 +0000 2023-09-30T00:46:06.447 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-09-30 00:40:28.516251133 +0000 2023-09-30T00:46:06.447 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-09-30T00:46:06.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:46:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-09-30T00:46:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-09-30T00:46:06.509 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000118273 s, 4.3 MB/s 2023-09-30T00:46:06.510 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:46:06.567 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2023-09-30T00:46:06.622 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:46:06.622 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 219289 Links: 1 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-09-30 00:45:53.711032043 +0000 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-09-30 00:40:28.999246543 +0000 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-09-30 00:40:28.999246543 +0000 2023-09-30T00:46:06.623 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-09-30T00:46:06.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:46:06.686 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-09-30T00:46:06.686 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-09-30T00:46:06.687 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000113133 s, 4.5 MB/s 2023-09-30T00:46:06.687 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:46:06.743 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2023-09-30T00:46:06.799 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:46:06.799 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 6h/6d Inode: 219000 Links: 1 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2023-09-30 00:45:53.711032043 +0000 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2023-09-30 00:40:29.493241850 +0000 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2023-09-30 00:40:29.493241850 +0000 2023-09-30T00:46:06.800 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2023-09-30T00:46:06.801 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:46:06.863 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2023-09-30T00:46:06.864 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2023-09-30T00:46:06.864 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.8605e-05 s, 5.2 MB/s 2023-09-30T00:46:06.865 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:46:06.921 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:06.921 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:06.921 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:46:06.936 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:46:06.936 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 218278 Links: 1 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-09-30 00:45:42.124195893 +0000 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-09-30 00:40:10.592974516 +0000 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-09-30 00:40:10.592974516 +0000 2023-09-30T00:46:06.992 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-09-30T00:46:06.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:46:07.054 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-09-30T00:46:07.054 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-09-30T00:46:07.055 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000234953 s, 2.2 MB/s 2023-09-30T00:46:07.055 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:46:07.111 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2023-09-30T00:46:07.166 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 220377 Links: 1 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-09-30 00:45:42.124195893 +0000 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-09-30 00:40:11.067966667 +0000 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-09-30 00:40:11.067966667 +0000 2023-09-30T00:46:07.167 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-09-30T00:46:07.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:46:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-09-30T00:46:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-09-30T00:46:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000241832 s, 2.1 MB/s 2023-09-30T00:46:07.231 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:46:07.286 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2023-09-30T00:46:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:46:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:07.341 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 213732 Links: 1 2023-09-30T00:46:07.342 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:07.342 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:07.342 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-09-30 00:45:42.124195893 +0000 2023-09-30T00:46:07.342 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-09-30 00:40:11.570958354 +0000 2023-09-30T00:46:07.342 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-09-30 00:40:11.570958354 +0000 2023-09-30T00:46:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-09-30T00:46:07.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:46:07.404 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-09-30T00:46:07.404 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-09-30T00:46:07.405 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000224737 s, 2.3 MB/s 2023-09-30T00:46:07.405 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:46:07.461 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2023-09-30T00:46:07.516 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:46:07.516 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:46:07.516 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 217912 Links: 1 2023-09-30T00:46:07.517 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:46:07.517 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:46:07.517 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-09-30 00:45:42.124195893 +0000 2023-09-30T00:46:07.517 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-09-30 00:40:12.069950108 +0000 2023-09-30T00:46:07.517 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-09-30 00:40:12.069950108 +0000 2023-09-30T00:46:07.517 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-09-30T00:46:07.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:46:07.579 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-09-30T00:46:07.579 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-09-30T00:46:07.579 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00019794 s, 2.6 MB/s 2023-09-30T00:46:07.580 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:46:07.636 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:07.636 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:46:07.636 INFO:tasks.ceph:Generating config... 2023-09-30T00:46:07.637 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T00:46:07.637 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T00:46:07.637 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-09-30T00:46:07.637 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] osd backoff on peering = True 2023-09-30T00:46:07.638 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd max scrubs = 2 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T00:46:07.639 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2023-09-30T00:46:07.640 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] ms type = async 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-09-30T00:46:07.640 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2023-09-30T00:46:07.640 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:46:07.640 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:46:07.640 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:46:07.640 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:46:07.641 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:46:07.641 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T00:46:07.641 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:46:07.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:07.716 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:46:07.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:46:07.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:46:07.827 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.17'), ('mon.c', '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]'), ('mon.b', '172.21.15.71')] 2023-09-30T00:46:07.827 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.71', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T00:46:07.828 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:07.828 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:46:07.882 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.17 --addv c '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]' --add b 172.21.15.71 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:07.965 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid 68e21822-bcf9-49ad-ac35-975ce04d146e 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = octopus 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:fsid 68e21822-bcf9-49ad-ac35-975ce04d146e 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2023-09-30T00:46:07.966347+0000 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:created 2023-09-30T00:46:07.966347+0000 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 15 (octopus) 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:election_strategy: 1 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2023-09-30T00:46:07.966 INFO:teuthology.orchestra.run.smithi017.stdout:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2023-09-30T00:46:07.967 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T00:46:07.968 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:46:08.023 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 68e21822-bcf9-49ad-ac35-975ce04d146e... 2023-09-30T00:46:08.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:46:08.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:46:08.107 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2023-09-30T00:46:08.107 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2023-09-30T00:46:08.107 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:46:08.107 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.108 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:46:08.109 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2023-09-30T00:46:08.110 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.71" 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: mon client directed command retry = 5 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: mon election default strategy = 1 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: ms bind msgr1 = False 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: ms bind msgr2 = True 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject socket failures = 5000 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: ms type = async 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: osd_async_recovery_min_cost = 1 2023-09-30T00:46:08.111 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_min_size = 2 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_size = 2 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 68e21822-bcf9-49ad-ac35-975ce04d146e 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2023-09-30T00:46:08.112 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.113 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: debug monc = 20 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore block size = 96636764160 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore compression algorithm = lz4 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore compression mode = aggressive 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore fsck on mount = True 2023-09-30T00:46:08.114 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore zero block detection = True 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluefs = 20 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluestore = 20 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 20 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: debug rocksdb = 10 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd full ratio = 0.9 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: osd backoff on peering = True 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify cached snaps = True 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify missing on start = True 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:46:08.115 INFO:teuthology.orchestra.run.smithi017.stdout: osd heartbeat use min delay socket = True 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd inject bad map crc probability = 0.1 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd max markdown count = 1000 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd max scrubs = 2 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock override recovery settings = True 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd objectstore = bluestore 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd op queue = debug_random 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd op queue cut off = debug_random 2023-09-30T00:46:08.116 INFO:teuthology.orchestra.run.smithi017.stdout: osd shutdown pgref assert = True 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout: osd_max_pg_log_entries = 3000 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout: osd_min_pg_log_entries = 3000 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2023-09-30T00:46:08.118 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2023-09-30T00:46:08.119 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:46:08.120 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: # 1m isn't quite enough 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: mon scrub interval = 300 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:46:08.121 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2023-09-30T00:46:08.122 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.c] 2023-09-30T00:46:08.122 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.b] 2023-09-30T00:46:08.123 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-09-30T00:46:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-09-30T00:46:08.123 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-09-30T00:46:08.124 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:46:08.125 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.126 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.71" 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 1 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: ms bind msgr1 = False 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: ms bind msgr2 = True 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 5000 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2023-09-30T00:46:08.127 INFO:teuthology.orchestra.run.smithi071.stdout: osd_async_recovery_min_cost = 1 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_min_size = 2 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_size = 2 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 68e21822-bcf9-49ad-ac35-975ce04d146e 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-09-30T00:46:08.128 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-09-30T00:46:08.129 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = lz4 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2023-09-30T00:46:08.130 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: osd backoff on peering = True 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2023-09-30T00:46:08.131 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd heartbeat use min delay socket = True 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd inject bad map crc probability = 0.1 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd max markdown count = 1000 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd max scrubs = 2 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_pg_log_entries = 3000 2023-09-30T00:46:08.132 INFO:teuthology.orchestra.run.smithi071.stdout: osd_min_pg_log_entries = 3000 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-09-30T00:46:08.133 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:46:08.134 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-09-30T00:46:08.135 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2023-09-30T00:46:08.136 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2023-09-30T00:46:08.136 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:46:08.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:46:08.215 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:46:08.215 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:08.215 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:46:08.230 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:08.230 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:46:08.285 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:46:08.285 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:08.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:46:08.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:46:08.361 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:08.361 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:08.416 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:46:08.416 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:08.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:46:08.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:46:08.450 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:08.451 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:08.505 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:46:08.505 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:46:08.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:46:08.556 INFO:teuthology.orchestra.run.smithi017.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:46:08.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:46:08.615 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:46:08.618 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:46:08.618 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:46:08.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:46:08.657 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:46:08.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T00:46:08.708 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T00:46:08.720 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:46:08.720 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:46:08.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:46:08.745 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:08.746 INFO:tasks.ceph:role: osd.0 2023-09-30T00:46:08.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:46:08.746 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:46:08.810 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:08.811 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:08.833 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2023-09-30T00:46:08.835 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-09-30T00:46:08.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:46:08.913 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:46:08.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:46:09.050 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:09.050 INFO:tasks.ceph:role: osd.1 2023-09-30T00:46:09.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:46:09.050 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:46:09.114 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:09.115 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:09.137 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2023-09-30T00:46:09.139 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-09-30T00:46:09.139 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T00:46:09.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:46:09.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:46:09.352 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:09.352 INFO:tasks.ceph:role: osd.2 2023-09-30T00:46:09.352 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:46:09.352 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:09.417 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:09.418 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:09.418 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:09.440 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2023-09-30T00:46:09.442 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-09-30T00:46:09.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T00:46:09.519 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:46:09.588 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:46:09.653 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:09.653 INFO:tasks.ceph:role: osd.3 2023-09-30T00:46:09.653 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:46:09.653 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:46:09.718 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:09.718 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:09.718 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:09.719 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:09.741 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2023-09-30T00:46:09.743 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2023-09-30T00:46:09.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T00:46:09.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:46:09.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:09.982 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:09.981+0000 7f9d0353f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:46:09.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:09.981+0000 7f9d0353f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:46:09.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:09.981+0000 7f9d0353f540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:46:09.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:09.981+0000 7f9d0353f540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:46:09.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:09.981+0000 7f9d0353f540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:46:09.983 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:09.981+0000 7f9d0353f540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T00:46:10.503 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:10.502+0000 7f9d0353f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:11.265 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:11.264+0000 7f9d0353f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:12.029 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.027+0000 7f9d0353f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:12.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:46:12.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:12.943 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.941+0000 7f21cf2eb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:46:12.943 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.941+0000 7f21cf2eb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:46:12.943 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.941+0000 7f21cf2eb540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:46:12.943 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.941+0000 7f21cf2eb540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:46:12.943 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.941+0000 7f21cf2eb540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:46:12.944 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:12.942+0000 7f21cf2eb540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T00:46:13.465 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:13.463+0000 7f21cf2eb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:14.234 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:14.232+0000 7f21cf2eb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:14.996 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:14.995+0000 7f21cf2eb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:15.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:46:15.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:15.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:15.909+0000 7fe4ad366540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:46:15.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:15.909+0000 7fe4ad366540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:46:15.911 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:15.910+0000 7fe4ad366540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:46:15.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:15.910+0000 7fe4ad366540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:46:15.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:15.910+0000 7fe4ad366540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:46:15.912 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:15.910+0000 7fe4ad366540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T00:46:16.437 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:16.435+0000 7fe4ad366540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:17.202 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:17.201+0000 7fe4ad366540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:17.965 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:17.963+0000 7fe4ad366540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:18.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:46:18.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:18.875+0000 7f288acdb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:46:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:18.875+0000 7f288acdb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:46:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:18.875+0000 7f288acdb540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:46:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:18.875+0000 7f288acdb540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:46:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:18.875+0000 7f288acdb540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:46:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:18.875+0000 7f288acdb540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T00:46:19.402 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:19.401+0000 7f288acdb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:20.167 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:20.166+0000 7f288acdb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:20.932 INFO:teuthology.orchestra.run.smithi017.stderr:2023-09-30T00:46:20.931+0000 7f288acdb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:21.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:46:21.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:21.784 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:21.784 INFO:tasks.ceph:role: osd.4 2023-09-30T00:46:21.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:46:21.784 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:46:21.860 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:21.860 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:21.860 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:21.860 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 2023-09-30T00:46:21.861 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:21.861 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:21.861 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:21.861 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:21.861 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:21.861 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:22.290 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-09-30T00:46:22.292 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-09-30T00:46:22.293 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:22.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:22.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:22.546 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:22.546 INFO:tasks.ceph:role: osd.5 2023-09-30T00:46:22.546 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:46:22.547 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:46:22.618 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:22.618 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:22.618 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:22.618 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 2023-09-30T00:46:22.618 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:22.619 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:22.619 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:22.619 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:22.619 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:22.619 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:23.045 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-09-30T00:46:23.047 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-09-30T00:46:23.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:23.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:23.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:23.262 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:23.263 INFO:tasks.ceph:role: osd.6 2023-09-30T00:46:23.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:46:23.263 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:46:23.334 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:23.334 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:23.334 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:23.334 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 2023-09-30T00:46:23.335 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:23.335 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:23.335 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:23.335 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:23.335 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:23.335 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:23.788 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-09-30T00:46:23.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-09-30T00:46:23.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:23.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:23.924 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:23.988 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:23.989 INFO:tasks.ceph:role: osd.7 2023-09-30T00:46:23.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:46:23.989 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:24.061 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:24.062 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:24.062 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:24.062 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:24.520 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-09-30T00:46:24.522 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-09-30T00:46:24.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:24.682 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:24.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:24.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:24.807+0000 7efde39ff540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T00:46:24.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:24.810+0000 7efde39ff540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T00:46:24.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:24.810+0000 7efde39ff540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:46:24.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:24.810+0000 7efde39ff540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:46:24.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:24.810+0000 7efde39ff540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:46:24.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:24.812+0000 7efde39ff540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T00:46:25.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:25.338+0000 7efde39ff540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:26.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:26.103+0000 7efde39ff540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:26.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:26.865+0000 7efde39ff540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:27.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:27.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:27.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:27.772+0000 7f4003c60540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T00:46:27.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:27.773+0000 7f4003c60540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T00:46:27.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:27.773+0000 7f4003c60540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:46:27.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:27.773+0000 7f4003c60540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:46:27.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:27.773+0000 7f4003c60540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:46:27.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:27.773+0000 7f4003c60540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T00:46:28.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:28.295+0000 7f4003c60540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:29.057+0000 7f4003c60540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:29.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:29.820+0000 7f4003c60540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:30.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:30.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:30.763+0000 7f2dcc9f0540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T00:46:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:30.763+0000 7f2dcc9f0540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T00:46:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:30.763+0000 7f2dcc9f0540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:46:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:30.763+0000 7f2dcc9f0540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:46:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:30.763+0000 7f2dcc9f0540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:46:30.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:30.763+0000 7f2dcc9f0540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T00:46:31.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:31.286+0000 7f2dcc9f0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:32.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:32.051+0000 7f2dcc9f0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:32.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:32.813+0000 7f2dcc9f0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:33.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:33.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:33.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:33.749+0000 7f065174b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T00:46:33.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:33.749+0000 7f065174b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T00:46:33.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:33.749+0000 7f065174b540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:46:33.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:33.749+0000 7f065174b540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:46:33.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:33.749+0000 7f065174b540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:46:33.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:33.749+0000 7f065174b540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T00:46:34.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:34.275+0000 7f065174b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:35.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:35.040+0000 7f065174b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-09-30T00:46:35.811+0000 7f065174b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:36.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:36.666 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:46:36.667 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:36.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T00:46:36.698 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:36.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:46:36.768 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:36.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:46:36.838 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:36.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:46:36.906 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:36.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:46:36.973 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:36.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:46:36.997 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:36.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T00:46:37.062 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:37.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T00:46:37.125 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:37.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T00:46:37.189 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:37.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T00:46:37.253 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:37.253 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:46:37.270 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:37.270 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T00:46:37.308 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:46:37.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:46:37.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:46:37.351 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.y] 2023-09-30T00:46:37.351 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDQbxdl0rAsIRAAc43oZeUdms0zQ1TFaiGi2Q== 2023-09-30T00:46:37.351 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2023-09-30T00:46:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDRbxdlz1+VOhAAyr/n5hYZFMlmNKj+h1RL7A== 2023-09-30T00:46:37.352 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2023-09-30T00:46:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDUbxdlze05OBAATK3SGU6oyBm4PwzUmJ/dRg== 2023-09-30T00:46:37.352 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2023-09-30T00:46:37.352 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDXbxdlykJVNhAAdra/ITnnkffgACp+bR/x+A== 2023-09-30T00:46:37.353 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.3] 2023-09-30T00:46:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDabxdlM/JDNBAACrQmq4ZiWKJ8CAu72hs0dw== 2023-09-30T00:46:37.353 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2023-09-30T00:46:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDQbxdlbI64JBAA9x4mLCSPdDEo0ieLsv3XaA== 2023-09-30T00:46:37.353 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.4] 2023-09-30T00:46:37.353 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDgbxdl8vA5MBAAPdYtRUUszSy8OOuB0prj7w== 2023-09-30T00:46:37.354 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.5] 2023-09-30T00:46:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDjbxdl020kLhAAzPG0sxViWh7jJZxWnXpAmw== 2023-09-30T00:46:37.354 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.6] 2023-09-30T00:46:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDmbxdlBKSOLRAAS6cWz0tq4TR+jBCsDTKDzA== 2023-09-30T00:46:37.354 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.7] 2023-09-30T00:46:37.354 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDpbxdlGVG6LBAA/cTL0aDmsPKo7RptJREB9A== 2023-09-30T00:46:37.355 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2023-09-30T00:46:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDQbxdlupY6JxAATtyGZYBfsDx+8C/ttycBiQ== 2023-09-30T00:46:37.355 INFO:teuthology.orchestra.run.smithi017.stdout:[client.1] 2023-09-30T00:46:37.355 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDQbxdlmFlDKhAAHUH/eFOkghO915w/MDZB+A== 2023-09-30T00:46:37.370 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2023-09-30T00:46:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDQbxdl0rAsIRAAc43oZeUdms0zQ1TFaiGi2Q== 2023-09-30T00:46:37.370 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-09-30T00:46:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDRbxdlz1+VOhAAyr/n5hYZFMlmNKj+h1RL7A== 2023-09-30T00:46:37.370 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-09-30T00:46:37.370 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDUbxdlze05OBAATK3SGU6oyBm4PwzUmJ/dRg== 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDXbxdlykJVNhAAdra/ITnnkffgACp+bR/x+A== 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDabxdlM/JDNBAACrQmq4ZiWKJ8CAu72hs0dw== 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDQbxdlbI64JBAA9x4mLCSPdDEo0ieLsv3XaA== 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDgbxdl8vA5MBAAPdYtRUUszSy8OOuB0prj7w== 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDjbxdl020kLhAAzPG0sxViWh7jJZxWnXpAmw== 2023-09-30T00:46:37.371 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDmbxdlBKSOLRAAS6cWz0tq4TR+jBCsDTKDzA== 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDpbxdlGVG6LBAA/cTL0aDmsPKo7RptJREB9A== 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDQbxdlupY6JxAATtyGZYBfsDx+8C/ttycBiQ== 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2023-09-30T00:46:37.372 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDQbxdlmFlDKhAAHUH/eFOkghO915w/MDZB+A== 2023-09-30T00:46:37.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:37.398 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:37.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.511 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.609 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:37.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:37.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.854 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:37.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:37.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:37.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:37.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:38.009 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:46:38.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:46:38.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:38.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:46:38.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T00:46:38.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:38.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T00:46:38.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T00:46:38.361 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:38.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T00:46:38.475 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:38.478 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:38.529 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:46:38.530 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:46:38.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T00:46:38.533 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:46:38.533 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T00:46:38.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T00:46:38.535 INFO:tasks.ceph.mon.c:Started 2023-09-30T00:46:38.535 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T00:46:38.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T00:46:38.572 INFO:tasks.ceph.mon.b:Started 2023-09-30T00:46:38.572 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:46:38.572 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T00:46:38.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T00:46:38.575 INFO:tasks.ceph.mgr.y:Started 2023-09-30T00:46:38.575 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:46:38.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T00:46:38.576 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:46:38.577 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:46:38.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:46:44.834 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2023-09-30T00:46:44.845 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:46:44.846 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:44.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:46:44.872 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:44.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:46:44.937 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:44.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:46:45.002 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2023-09-30T00:46:45.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:46:45.068 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:45.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T00:46:45.093 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:45.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T00:46:45.157 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:45.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T00:46:45.162 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.162+0000 7ffb69c77180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:46:45.220 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-09-30T00:46:45.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T00:46:45.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new e0fd89fc-05f1-4087-a5e3-1522fb41339b 0 2023-09-30T00:46:45.300 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.299+0000 7ffb69c77180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:45.378 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.378+0000 7ffb69c77180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:46:45.534 INFO:teuthology.orchestra.run.smithi071.stdout:0 2023-09-30T00:46:45.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 320ee357-49bb-4a6b-9676-220ea8d867a5 1 2023-09-30T00:46:45.570 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.570+0000 7ffb69c77180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:45.750 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.750+0000 7ffb69c77180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:46:45.778 INFO:teuthology.orchestra.run.smithi071.stdout:1 2023-09-30T00:46:45.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 07c3c896-3c42-4840-8824-ab3e35142c56 2 2023-09-30T00:46:45.831 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.831+0000 7ffb69c77180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:46:45.855 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:45.855+0000 7fa517454180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:46:45.971 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:45.971+0000 7ffb69c77180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:46:45.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:45.992+0000 7fa517454180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:46.003 INFO:teuthology.orchestra.run.smithi071.stdout:2 2023-09-30T00:46:46.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 320b1738-9a22-49a6-bb8d-0c5cf165571a 3 2023-09-30T00:46:46.071 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:46.070+0000 7ffb69c77180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:46.073 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:46.073+0000 7fa517454180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:46:46.226 INFO:teuthology.orchestra.run.smithi071.stdout:3 2023-09-30T00:46:46.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 3a3ae6c2-5973-400f-9d51-0a3a896b225d 4 2023-09-30T00:46:46.265 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:46.264+0000 7fa517454180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:46.444 INFO:teuthology.orchestra.run.smithi071.stdout:4 2023-09-30T00:46:46.445 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:46.445+0000 7fa517454180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:46:46.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new a6c34efb-477e-4157-ac55-6f93592ff787 5 2023-09-30T00:46:46.526 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:46.526+0000 7fa517454180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:46:46.606 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:46.605+0000 7ffb69c77180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:46:46.666 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:46.666+0000 7fa517454180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:46:46.687 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:46.687+0000 7ffb69c77180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:46:46.707 INFO:teuthology.orchestra.run.smithi071.stdout:5 2023-09-30T00:46:46.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 8ed0a2c2-6dae-42b5-ab67-a2fac8753605 6 2023-09-30T00:46:46.765 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:46.765+0000 7fa517454180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:46.768 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:46.768+0000 7ffb69c77180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:46:46.967 INFO:teuthology.orchestra.run.smithi071.stdout:6 2023-09-30T00:46:46.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new a2d20a42-04a9-4102-af3d-5ed71c587ab1 7 2023-09-30T00:46:47.234 INFO:teuthology.orchestra.run.smithi071.stdout:7 2023-09-30T00:46:47.251 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:46:47.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T00:46:47.259 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:46:47.260 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:46:47.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T00:46:47.265 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:46:47.265 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:46:47.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T00:46:47.267 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:46:47.268 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:46:47.268 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T00:46:47.272 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:46:47.272 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T00:46:47.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T00:46:47.274 INFO:tasks.ceph.osd.4:Started 2023-09-30T00:46:47.275 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T00:46:47.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T00:46:47.277 INFO:tasks.ceph.osd.5:Started 2023-09-30T00:46:47.277 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T00:46:47.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-30T00:46:47.279 INFO:tasks.ceph.osd.6:Started 2023-09-30T00:46:47.279 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T00:46:47.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-30T00:46:47.281 INFO:tasks.ceph.osd.7:Started 2023-09-30T00:46:47.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:47.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:47.326+0000 7fa517454180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:46:47.412 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:47.412+0000 7ffb69c77180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:46:47.418 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:47.419+0000 7fa517454180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:46:47.496 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:47.497+0000 7fa517454180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:46:47.508 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:47.508 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":10,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:47.228732+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:47.521 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:46:47.521 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:46:47.559 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:47.558+0000 7ffb69c77180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:47.664 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:47.663+0000 7ffb69c77180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:46:47.874 INFO:tasks.ceph.osd.5.smithi071.stderr:2023-09-30T00:46:47.874+0000 7f5a84ba8540 -1 Falling back to public interface 2023-09-30T00:46:47.876 INFO:tasks.ceph.osd.0.smithi017.stderr:2023-09-30T00:46:47.876+0000 7ff9686ec540 -1 Falling back to public interface 2023-09-30T00:46:47.877 INFO:tasks.ceph.osd.6.smithi071.stderr:2023-09-30T00:46:47.878+0000 7f7107ad1540 -1 Falling back to public interface 2023-09-30T00:46:47.878 INFO:tasks.ceph.osd.7.smithi071.stderr:2023-09-30T00:46:47.878+0000 7ff5c3b76540 -1 Falling back to public interface 2023-09-30T00:46:47.886 INFO:tasks.ceph.osd.4.smithi071.stderr:2023-09-30T00:46:47.886+0000 7fe54be82540 -1 Falling back to public interface 2023-09-30T00:46:47.888 INFO:tasks.ceph.osd.2.smithi017.stderr:2023-09-30T00:46:47.888+0000 7feb2dfb6540 -1 Falling back to public interface 2023-09-30T00:46:47.892 INFO:tasks.ceph.osd.1.smithi017.stderr:2023-09-30T00:46:47.893+0000 7f46790df540 -1 Falling back to public interface 2023-09-30T00:46:47.900 INFO:tasks.ceph.osd.3.smithi017.stderr:2023-09-30T00:46:47.900+0000 7f75f2191540 -1 Falling back to public interface 2023-09-30T00:46:47.921 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:47.922+0000 7ffb69c77180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:46:48.002 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:48.002+0000 7ffb69c77180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:46:48.067 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:48.068+0000 7fa517454180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:46:48.206 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:48.206+0000 7fa517454180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:48.291 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:48.292+0000 7fa517454180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:46:48.327 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:48.517 INFO:teuthology.misc.health.smithi017.stdout: 2023-09-30T00:46:48.517 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":10,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:47.228732+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:48.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:46:48.538 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:48.538+0000 7fa517454180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:46:48.615 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:48.615+0000 7fa517454180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:46:48.869 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:48.869+0000 7ffb69c77180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:46:48.947 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:48.946+0000 7ffb69c77180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:46:49.024 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.024+0000 7ffb69c77180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:46:49.164 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.164+0000 7ffb69c77180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:46:49.251 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.251+0000 7ffb69c77180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:46:49.410 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.410+0000 7ffb69c77180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:46:49.441 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:49.442+0000 7fa517454180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:46:49.490 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.489+0000 7ffb69c77180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:46:49.514 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:49.515+0000 7fa517454180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:46:49.568 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.568+0000 7ffb69c77180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:46:49.590 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:49.590+0000 7fa517454180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:46:49.647 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.647+0000 7ffb69c77180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:46:49.691 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.691+0000 7ffaf9f6c700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:46:49.691 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.691+0000 7ffaf9f6c700 -1 pg_autoscaler.serve: 2023-09-30T00:46:49.692 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:49.691+0000 7ffaf9f6c700 -1 Traceback (most recent call last): 2023-09-30T00:46:49.692 INFO:tasks.ceph.mgr.y.smithi017.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:46:49.692 INFO:tasks.ceph.mgr.y.smithi017.stderr: self._maybe_adjust() 2023-09-30T00:46:49.693 INFO:tasks.ceph.mgr.y.smithi017.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:46:49.693 INFO:tasks.ceph.mgr.y.smithi017.stderr: if self.noautoscale: 2023-09-30T00:46:49.693 INFO:tasks.ceph.mgr.y.smithi017.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:46:49.693 INFO:tasks.ceph.mgr.y.smithi017.stderr: 2023-09-30T00:46:49.725 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:49.726+0000 7fa517454180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:46:49.810 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:49.810+0000 7fa517454180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:46:49.964 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:49.964+0000 7fa517454180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:46:50.041 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:50.041+0000 7fa517454180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:46:50.117 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:50.117+0000 7fa517454180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:46:50.193 INFO:tasks.ceph.mgr.x.smithi071.stderr:2023-09-30T00:46:50.193+0000 7fa517454180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:46:50.677 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:50.676+0000 7ffb2307d700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T00:46:50.681 INFO:tasks.ceph.mgr.y.smithi017.stderr:2023-09-30T00:46:50.681+0000 7ffb2307d700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T00:46:51.366 INFO:tasks.ceph.osd.0.smithi017.stderr:2023-09-30T00:46:51.365+0000 7ff9686ec540 -1 osd.0 0 log_to_monitors true 2023-09-30T00:46:51.409 INFO:tasks.ceph.osd.4.smithi071.stderr:2023-09-30T00:46:51.410+0000 7fe54be82540 -1 osd.4 0 log_to_monitors true 2023-09-30T00:46:51.421 INFO:tasks.ceph.osd.2.smithi017.stderr:2023-09-30T00:46:51.421+0000 7feb2dfb6540 -1 osd.2 0 log_to_monitors true 2023-09-30T00:46:51.426 INFO:tasks.ceph.osd.3.smithi017.stderr:2023-09-30T00:46:51.426+0000 7f75f2191540 -1 osd.3 0 log_to_monitors true 2023-09-30T00:46:51.430 INFO:tasks.ceph.osd.1.smithi017.stderr:2023-09-30T00:46:51.430+0000 7f46790df540 -1 osd.1 0 log_to_monitors true 2023-09-30T00:46:51.444 INFO:tasks.ceph.osd.5.smithi071.stderr:2023-09-30T00:46:51.445+0000 7f5a84ba8540 -1 osd.5 0 log_to_monitors true 2023-09-30T00:46:51.453 INFO:tasks.ceph.osd.6.smithi071.stderr:2023-09-30T00:46:51.453+0000 7f7107ad1540 -1 osd.6 0 log_to_monitors true 2023-09-30T00:46:51.458 INFO:tasks.ceph.osd.7.smithi071.stderr:2023-09-30T00:46:51.459+0000 7ff5c3b76540 -1 osd.7 0 log_to_monitors true 2023-09-30T00:46:52.914 INFO:tasks.ceph.osd.2.smithi017.stderr:2023-09-30T00:46:52.914+0000 7feb24bd7700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:46:52.915 INFO:tasks.ceph.osd.1.smithi017.stderr:2023-09-30T00:46:52.915+0000 7f466fd00700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:46:52.916 INFO:tasks.ceph.osd.4.smithi071.stderr:2023-09-30T00:46:52.916+0000 7fe542aa3700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T00:46:52.916 INFO:tasks.ceph.osd.3.smithi017.stderr:2023-09-30T00:46:52.914+0000 7f75e8db2700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:46:52.916 INFO:tasks.ceph.osd.7.smithi071.stderr:2023-09-30T00:46:52.916+0000 7ff5ba797700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T00:46:52.917 INFO:tasks.ceph.osd.5.smithi071.stderr:2023-09-30T00:46:52.916+0000 7f5a7b7c9700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T00:46:52.917 INFO:tasks.ceph.osd.6.smithi071.stderr:2023-09-30T00:46:52.916+0000 7f70fe6f2700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T00:46:52.918 INFO:tasks.ceph.osd.6.smithi071.stderr:2023-09-30T00:46:52.917+0000 7f70f3e8a700 -1 osd.6 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.918 INFO:tasks.ceph.osd.4.smithi071.stderr:2023-09-30T00:46:52.917+0000 7fe53823b700 -1 osd.4 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.918 INFO:tasks.ceph.osd.2.smithi017.stderr:2023-09-30T00:46:52.917+0000 7feb1a36f700 -1 osd.2 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.919 INFO:tasks.ceph.osd.3.smithi017.stderr:2023-09-30T00:46:52.917+0000 7f75de54a700 -1 osd.3 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.919 INFO:tasks.ceph.osd.1.smithi017.stderr:2023-09-30T00:46:52.918+0000 7f4665498700 -1 osd.1 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.920 INFO:tasks.ceph.osd.7.smithi071.stderr:2023-09-30T00:46:52.917+0000 7ff5aff2f700 -1 osd.7 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.920 INFO:tasks.ceph.osd.0.smithi017.stderr:2023-09-30T00:46:52.920+0000 7ff95f30d700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:46:52.921 INFO:tasks.ceph.osd.4.smithi071.stderr:2023-09-30T00:46:52.922+0000 7fe53d09d700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.922 INFO:tasks.ceph.osd.6.smithi071.stderr:2023-09-30T00:46:52.921+0000 7f70f8cec700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.922 INFO:tasks.ceph.osd.5.smithi071.stderr:2023-09-30T00:46:52.922+0000 7f5a75dc3700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.922 INFO:tasks.ceph.osd.7.smithi071.stderr:2023-09-30T00:46:52.922+0000 7ff5b4d91700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.923 INFO:tasks.ceph.osd.2.smithi017.stderr:2023-09-30T00:46:52.922+0000 7feb1f1d1700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.923 INFO:tasks.ceph.osd.1.smithi017.stderr:2023-09-30T00:46:52.923+0000 7f466a2fa700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.924 INFO:tasks.ceph.osd.3.smithi017.stderr:2023-09-30T00:46:52.923+0000 7f75e33ac700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:52.924 INFO:tasks.ceph.osd.0.smithi017.stderr:2023-09-30T00:46:52.921+0000 7ff954aa5700 -1 osd.0 0 handle_osd_map injecting map crc failure 2023-09-30T00:46:52.926 INFO:tasks.ceph.osd.0.smithi017.stderr:2023-09-30T00:46:52.926+0000 7ff959907700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:46:54.917 INFO:tasks.ceph.mon.a.smithi017.stderr:2023-09-30T00:46:54.917+0000 7f74e53d8700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:46:55.335 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:55.668 INFO:teuthology.misc.health.smithi017.stdout: 2023-09-30T00:46:55.668 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":14,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:54.922616+0000","last_up_change":"2023-09-30T00:46:53.912335+0000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":88142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6801","nonce":88142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":88142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":88142}]},"public_addr":"172.21.15.17:6800/88142","cluster_addr":"172.21.15.17:6801/88142","heartbeat_back_addr":"172.21.15.17:6803/88142","heartbeat_front_addr":"172.21.15.17:6802/88142","state":["exists","up"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":88158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6809","nonce":88158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":88158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":88158}]},"public_addr":"172.21.15.17:6808/88158","cluster_addr":"172.21.15.17:6809/88158","heartbeat_back_addr":"172.21.15.17:6811/88158","heartbeat_front_addr":"172.21.15.17:6810/88158","state":["exists","up"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":88143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6805","nonce":88143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":88143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":88143}]},"public_addr":"172.21.15.17:6804/88143","cluster_addr":"172.21.15.17:6805/88143","heartbeat_back_addr":"172.21.15.17:6807/88143","heartbeat_front_addr":"172.21.15.17:6806/88143","state":["exists","up"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":88159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":88159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6815","nonce":88159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":88159}]},"public_addr":"172.21.15.17:6812/88159","cluster_addr":"172.21.15.17:6813/88159","heartbeat_back_addr":"172.21.15.17:6815/88159","heartbeat_front_addr":"172.21.15.17:6814/88159","state":["exists","up"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":88085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6813","nonce":88085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":88085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":88085}]},"public_addr":"172.21.15.71:6812/88085","cluster_addr":"172.21.15.71:6813/88085","heartbeat_back_addr":"172.21.15.71:6815/88085","heartbeat_front_addr":"172.21.15.71:6814/88085","state":["exists","up"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":88070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":88070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":88070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":88070}]},"public_addr":"172.21.15.71:6800/88070","cluster_addr":"172.21.15.71:6801/88070","heartbeat_back_addr":"172.21.15.71:6803/88070","heartbeat_front_addr":"172.21.15.71:6802/88070","state":["exists","up"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":88071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":88071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":88071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":88071}]},"public_addr":"172.21.15.71:6804/88071","cluster_addr":"172.21.15.71:6805/88071","heartbeat_back_addr":"172.21.15.71:6807/88071","heartbeat_front_addr":"172.21.15.71:6806/88071","state":["exists","up"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":88072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":88072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6811","nonce":88072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":88072}]},"public_addr":"172.21.15.71:6808/88072","cluster_addr":"172.21.15.71:6809/88072","heartbeat_back_addr":"172.21.15.71:6811/88072","heartbeat_front_addr":"172.21.15.71:6810/88072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.325066+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.472475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391217+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.428015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.465329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.456368+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.493001+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:55.681 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T00:46:55.681 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:46:55.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:46:56.934 INFO:tasks.ceph.osd.2.smithi017.stderr:2023-09-30T00:46:56.933+0000 7feb1a36f700 -1 osd.2 14 handle_osd_map injecting map crc failure 2023-09-30T00:46:56.942 INFO:tasks.ceph.osd.6.smithi071.stderr:2023-09-30T00:46:56.942+0000 7f70f1e86700 -1 osd.6 14 handle_osd_map injecting map crc failure 2023-09-30T00:46:56.944 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'rbd' created 2023-09-30T00:46:56.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:46:57.986 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:46:58.001 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:46:58.001 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:46:58.001 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:46:58.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:46:58.002 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:46:58.389 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:58.402 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":4,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":87896}]},"active_addr":"172.21.15.17:6816/87896","active_change":"2023-09-30T00:46:49.661695+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3323082807}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2656863997}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2987475780}]}]} 2023-09-30T00:46:58.408 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:46:58.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:46:58.408 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:58.738 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:58.739 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":17,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:57.933687+0000","last_up_change":"2023-09-30T00:46:53.912335+0000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:46:55.722423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:46:56.041527+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":88142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6801","nonce":88142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":88142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":88142}]},"public_addr":"172.21.15.17:6800/88142","cluster_addr":"172.21.15.17:6801/88142","heartbeat_back_addr":"172.21.15.17:6803/88142","heartbeat_front_addr":"172.21.15.17:6802/88142","state":["exists","up"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":88158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6809","nonce":88158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":88158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":88158}]},"public_addr":"172.21.15.17:6808/88158","cluster_addr":"172.21.15.17:6809/88158","heartbeat_back_addr":"172.21.15.17:6811/88158","heartbeat_front_addr":"172.21.15.17:6810/88158","state":["exists","up"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":88143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6805","nonce":88143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":88143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":88143}]},"public_addr":"172.21.15.17:6804/88143","cluster_addr":"172.21.15.17:6805/88143","heartbeat_back_addr":"172.21.15.17:6807/88143","heartbeat_front_addr":"172.21.15.17:6806/88143","state":["exists","up"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":88159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":88159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6815","nonce":88159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":88159}]},"public_addr":"172.21.15.17:6812/88159","cluster_addr":"172.21.15.17:6813/88159","heartbeat_back_addr":"172.21.15.17:6815/88159","heartbeat_front_addr":"172.21.15.17:6814/88159","state":["exists","up"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":88085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6813","nonce":88085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":88085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":88085}]},"public_addr":"172.21.15.71:6812/88085","cluster_addr":"172.21.15.71:6813/88085","heartbeat_back_addr":"172.21.15.71:6815/88085","heartbeat_front_addr":"172.21.15.71:6814/88085","state":["exists","up"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":88070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":88070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":88070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":88070}]},"public_addr":"172.21.15.71:6800/88070","cluster_addr":"172.21.15.71:6801/88070","heartbeat_back_addr":"172.21.15.71:6803/88070","heartbeat_front_addr":"172.21.15.71:6802/88070","state":["exists","up"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":88071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":88071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":88071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":88071}]},"public_addr":"172.21.15.71:6804/88071","cluster_addr":"172.21.15.71:6805/88071","heartbeat_back_addr":"172.21.15.71:6807/88071","heartbeat_front_addr":"172.21.15.71:6806/88071","state":["exists","up"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":88072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":88072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6811","nonce":88072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":88072}]},"public_addr":"172.21.15.71:6808/88072","cluster_addr":"172.21.15.71:6809/88072","heartbeat_back_addr":"172.21.15.71:6811/88072","heartbeat_front_addr":"172.21.15.71:6810/88072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.325066+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.472475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391217+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.428015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.465329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.456368+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.493001+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:58.753 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:46:58.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:58.981 INFO:tasks.ceph.osd.0.smithi017.stderr:2023-09-30T00:46:58.981+0000 7ff954aa5700 -1 osd.0 17 handle_osd_map injecting map crc failure 2023-09-30T00:46:59.080 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:46:59.080 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":18,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:58.979992+0000","last_up_change":"2023-09-30T00:46:53.912335+0000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:46:55.722423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:46:56.041527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":88142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6801","nonce":88142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":88142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":88142}]},"public_addr":"172.21.15.17:6800/88142","cluster_addr":"172.21.15.17:6801/88142","heartbeat_back_addr":"172.21.15.17:6803/88142","heartbeat_front_addr":"172.21.15.17:6802/88142","state":["exists","up"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":88158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6809","nonce":88158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":88158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":88158}]},"public_addr":"172.21.15.17:6808/88158","cluster_addr":"172.21.15.17:6809/88158","heartbeat_back_addr":"172.21.15.17:6811/88158","heartbeat_front_addr":"172.21.15.17:6810/88158","state":["exists","up"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":88143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6805","nonce":88143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":88143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":88143}]},"public_addr":"172.21.15.17:6804/88143","cluster_addr":"172.21.15.17:6805/88143","heartbeat_back_addr":"172.21.15.17:6807/88143","heartbeat_front_addr":"172.21.15.17:6806/88143","state":["exists","up"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":88159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":88159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6815","nonce":88159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":88159}]},"public_addr":"172.21.15.17:6812/88159","cluster_addr":"172.21.15.17:6813/88159","heartbeat_back_addr":"172.21.15.17:6815/88159","heartbeat_front_addr":"172.21.15.17:6814/88159","state":["exists","up"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":88085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6813","nonce":88085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":88085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":88085}]},"public_addr":"172.21.15.71:6812/88085","cluster_addr":"172.21.15.71:6813/88085","heartbeat_back_addr":"172.21.15.71:6815/88085","heartbeat_front_addr":"172.21.15.71:6814/88085","state":["exists","up"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":88070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":88070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":88070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":88070}]},"public_addr":"172.21.15.71:6800/88070","cluster_addr":"172.21.15.71:6801/88070","heartbeat_back_addr":"172.21.15.71:6803/88070","heartbeat_front_addr":"172.21.15.71:6802/88070","state":["exists","up"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":88071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":88071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":88071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":88071}]},"public_addr":"172.21.15.71:6804/88071","cluster_addr":"172.21.15.71:6805/88071","heartbeat_back_addr":"172.21.15.71:6807/88071","heartbeat_front_addr":"172.21.15.71:6806/88071","state":["exists","up"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":88072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":88072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6811","nonce":88072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":88072}]},"public_addr":"172.21.15.71:6808/88072","cluster_addr":"172.21.15.71:6809/88072","heartbeat_back_addr":"172.21.15.71:6811/88072","heartbeat_front_addr":"172.21.15.71:6810/88072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.325066+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.472475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391217+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.428015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.465329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.456368+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.493001+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:59.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:46:59.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:46:59.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:46:59.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:46:59.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T00:46:59.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T00:46:59.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T00:46:59.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T00:46:59.378 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:46:59.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:46:59.382 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:46:59.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:46:59.384 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:46:59.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:46:59.385 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:46:59.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:46:59.392 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:46:59.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:46:59.393 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:46:59.394 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:46:59.398 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:46:59.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:46:59.402 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:46:59.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:46:59.973 INFO:teuthology.orchestra.run.smithi017.stdout:55834574849 2023-09-30T00:46:59.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.1 2023-09-30T00:46:59.998 INFO:teuthology.orchestra.run.smithi017.stdout:55834574849 2023-09-30T00:47:00.004 INFO:teuthology.orchestra.run.smithi017.stdout:55834574849 2023-09-30T00:47:00.007 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:47:00.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.6 2023-09-30T00:47:00.029 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.7 2023-09-30T00:47:00.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2023-09-30T00:47:00.032 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:00.037 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:47:00.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.0 2023-09-30T00:47:00.054 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:00.060 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:47:00.061 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:47:00.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2023-09-30T00:47:00.077 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:00.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2023-09-30T00:47:00.077 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:00.093 INFO:teuthology.orchestra.run.smithi017.stdout:55834574851 2023-09-30T00:47:00.108 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2023-09-30T00:47:00.108 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:00.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:47:01.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:47:01.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:47:01.344 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:47:01.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.1 2023-09-30T00:47:01.358 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:01.381 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:47:01.382 INFO:teuthology.orchestra.run.smithi017.stdout:55834574850 2023-09-30T00:47:01.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.6 2023-09-30T00:47:01.397 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:01.398 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.7 2023-09-30T00:47:01.398 DEBUG:teuthology.parallel:result is None 2023-09-30T00:47:01.398 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:47:01.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:47:01.757 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:01.759 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:47:01.771 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-09-30T00:47:01.668094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000425"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002488+0000","last_change":"2023-09-30T00:46:57.997529+0000","last_active":"2023-09-30T00:46:58.002488+0000","last_peered":"2023-09-30T00:46:58.002488+0000","last_clean":"2023-09-30T00:46:58.002488+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:46:58.002488+0000","last_undegraded":"2023-09-30T00:46:58.002488+0000","last_fullsized":"2023-09-30T00:46:58.002488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:17:07.145653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992796+0000","last_change":"2023-09-30T00:46:57.992123+0000","last_active":"2023-09-30T00:46:57.992796+0000","last_peered":"2023-09-30T00:46:57.992796+0000","last_clean":"2023-09-30T00:46:57.992796+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:46:57.992796+0000","last_undegraded":"2023-09-30T00:46:57.992796+0000","last_fullsized":"2023-09-30T00:46:57.992796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:54:28.140664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.999672+0000","last_change":"2023-09-30T00:46:57.994425+0000","last_active":"2023-09-30T00:46:57.999672+0000","last_peered":"2023-09-30T00:46:57.999672+0000","last_clean":"2023-09-30T00:46:57.999672+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:46:57.999672+0000","last_undegraded":"2023-09-30T00:46:57.999672+0000","last_fullsized":"2023-09-30T00:46:57.999672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:56:54.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002593+0000","last_change":"2023-09-30T00:46:57.997288+0000","last_active":"2023-09-30T00:46:58.002593+0000","last_peered":"2023-09-30T00:46:58.002593+0000","last_clean":"2023-09-30T00:46:58.002593+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:46:58.002593+0000","last_undegraded":"2023-09-30T00:46:58.002593+0000","last_fullsized":"2023-09-30T00:46:58.002593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:15:43.543535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.104195+0000","last_change":"2023-09-30T00:46:56.946070+0000","last_active":"2023-09-30T00:46:58.104195+0000","last_peered":"2023-09-30T00:46:58.104195+0000","last_clean":"2023-09-30T00:46:58.104195+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:46:58.104195+0000","last_undegraded":"2023-09-30T00:46:58.104195+0000","last_fullsized":"2023-09-30T00:46:58.104195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_clean_scrub_stamp":"2023-09-30T00:46:55.928441+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:15.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992211+0000","last_change":"2023-09-30T00:46:57.990797+0000","last_active":"2023-09-30T00:46:57.992211+0000","last_peered":"2023-09-30T00:46:57.992211+0000","last_clean":"2023-09-30T00:46:57.992211+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:46:57.992211+0000","last_undegraded":"2023-09-30T00:46:57.992211+0000","last_fullsized":"2023-09-30T00:46:57.992211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:27:46.015207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.531899+0000","last_change":"2023-09-30T00:46:58.531319+0000","last_active":"2023-09-30T00:46:58.531899+0000","last_peered":"2023-09-30T00:46:58.531899+0000","last_clean":"2023-09-30T00:46:58.531899+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:46:58.531899+0000","last_undegraded":"2023-09-30T00:46:58.531899+0000","last_fullsized":"2023-09-30T00:46:58.531899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.203428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.994043+0000","last_change":"2023-09-30T00:46:57.991533+0000","last_active":"2023-09-30T00:46:57.994043+0000","last_peered":"2023-09-30T00:46:57.994043+0000","last_clean":"2023-09-30T00:46:57.994043+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:46:57.994043+0000","last_undegraded":"2023-09-30T00:46:57.994043+0000","last_fullsized":"2023-09-30T00:46:57.994043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:49:05.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.986609+0000","last_change":"2023-09-30T00:46:57.985988+0000","last_active":"2023-09-30T00:46:57.986609+0000","last_peered":"2023-09-30T00:46:57.986609+0000","last_clean":"2023-09-30T00:46:57.986609+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:46:57.986609+0000","last_undegraded":"2023-09-30T00:46:57.986609+0000","last_fullsized":"2023-09-30T00:46:57.986609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:43:22.827084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:47:01.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:47:02.099 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:02.101 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:47:02.112 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-09-30T00:47:01.668094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000425"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002488+0000","last_change":"2023-09-30T00:46:57.997529+0000","last_active":"2023-09-30T00:46:58.002488+0000","last_peered":"2023-09-30T00:46:58.002488+0000","last_clean":"2023-09-30T00:46:58.002488+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:46:58.002488+0000","last_undegraded":"2023-09-30T00:46:58.002488+0000","last_fullsized":"2023-09-30T00:46:58.002488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:17:07.145653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992796+0000","last_change":"2023-09-30T00:46:57.992123+0000","last_active":"2023-09-30T00:46:57.992796+0000","last_peered":"2023-09-30T00:46:57.992796+0000","last_clean":"2023-09-30T00:46:57.992796+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:46:57.992796+0000","last_undegraded":"2023-09-30T00:46:57.992796+0000","last_fullsized":"2023-09-30T00:46:57.992796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:54:28.140664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.999672+0000","last_change":"2023-09-30T00:46:57.994425+0000","last_active":"2023-09-30T00:46:57.999672+0000","last_peered":"2023-09-30T00:46:57.999672+0000","last_clean":"2023-09-30T00:46:57.999672+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:46:57.999672+0000","last_undegraded":"2023-09-30T00:46:57.999672+0000","last_fullsized":"2023-09-30T00:46:57.999672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:56:54.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002593+0000","last_change":"2023-09-30T00:46:57.997288+0000","last_active":"2023-09-30T00:46:58.002593+0000","last_peered":"2023-09-30T00:46:58.002593+0000","last_clean":"2023-09-30T00:46:58.002593+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:46:58.002593+0000","last_undegraded":"2023-09-30T00:46:58.002593+0000","last_fullsized":"2023-09-30T00:46:58.002593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:15:43.543535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.104195+0000","last_change":"2023-09-30T00:46:56.946070+0000","last_active":"2023-09-30T00:46:58.104195+0000","last_peered":"2023-09-30T00:46:58.104195+0000","last_clean":"2023-09-30T00:46:58.104195+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:46:58.104195+0000","last_undegraded":"2023-09-30T00:46:58.104195+0000","last_fullsized":"2023-09-30T00:46:58.104195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_clean_scrub_stamp":"2023-09-30T00:46:55.928441+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:15.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992211+0000","last_change":"2023-09-30T00:46:57.990797+0000","last_active":"2023-09-30T00:46:57.992211+0000","last_peered":"2023-09-30T00:46:57.992211+0000","last_clean":"2023-09-30T00:46:57.992211+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:46:57.992211+0000","last_undegraded":"2023-09-30T00:46:57.992211+0000","last_fullsized":"2023-09-30T00:46:57.992211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:27:46.015207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.531899+0000","last_change":"2023-09-30T00:46:58.531319+0000","last_active":"2023-09-30T00:46:58.531899+0000","last_peered":"2023-09-30T00:46:58.531899+0000","last_clean":"2023-09-30T00:46:58.531899+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:46:58.531899+0000","last_undegraded":"2023-09-30T00:46:58.531899+0000","last_fullsized":"2023-09-30T00:46:58.531899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.203428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.994043+0000","last_change":"2023-09-30T00:46:57.991533+0000","last_active":"2023-09-30T00:46:57.994043+0000","last_peered":"2023-09-30T00:46:57.994043+0000","last_clean":"2023-09-30T00:46:57.994043+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:46:57.994043+0000","last_undegraded":"2023-09-30T00:46:57.994043+0000","last_fullsized":"2023-09-30T00:46:57.994043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:49:05.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.986609+0000","last_change":"2023-09-30T00:46:57.985988+0000","last_active":"2023-09-30T00:46:57.986609+0000","last_peered":"2023-09-30T00:46:57.986609+0000","last_clean":"2023-09-30T00:46:57.986609+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:46:57.986609+0000","last_undegraded":"2023-09-30T00:46:57.986609+0000","last_fullsized":"2023-09-30T00:46:57.986609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:43:22.827084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:47:02.114 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:47:02.114 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:47:02.114 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:47:02.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:02.480 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:02.480 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:05.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:05.875 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:05.875 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:08.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:09.270 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:09.270 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:12.284 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:12.665 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:12.666 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:15.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:16.061 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:16.061 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:19.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:19.455 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:19.455 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:22.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:22.848 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:22.848 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:25.863 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:26.259 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:26.259 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:29.274 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:29.652 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:29.652 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:32.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:33.047 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:36.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:36.444 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:36.444 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:39.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:39.839 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:39.839 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:42.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:43.231 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:43.232 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:46.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:46.630 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:49.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:50.025 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:50.025 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:53.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:53.418 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:53.418 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:56.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:47:56.819 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:59.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:00.211 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:03.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:03.603 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:03.603 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:06.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:07.000 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:07.001 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:10.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:10.395 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:10.395 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:13.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:13.790 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:16.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:17.185 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:17.185 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:20.200 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:20.582 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:23.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:23.991 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:23.992 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:27.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:27.383 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:27.384 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:30.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:30.778 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:30.778 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:33.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:34.170 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:34.170 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:37.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:37.563 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:37.563 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:40.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:40.959 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:43.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:44.352 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:44.353 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:47.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:47.745 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:47.745 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:50.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:51.137 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:54.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:54.531 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:54.531 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:57.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:57.923 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:48:57.923 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:00.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:01.315 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:01.315 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:04.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:04.706 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:04.707 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:07.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:08.101 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:08.102 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:11.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:11.495 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:11.495 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:14.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:14.892 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:14.892 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:17.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:18.284 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:21.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:21.676 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:21.676 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:24.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:25.070 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:25.070 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:28.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:28.461 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:28.461 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:31.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:31.855 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:31.856 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:34.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:35.247 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:35.247 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:38.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:38.637 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:38.637 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:41.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:42.033 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:45.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:45.426 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:45.427 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:48.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:48.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:48.823 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:51.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:52.216 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:52.216 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:55.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:55.609 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:55.609 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:58.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:59.002 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:49:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:59.999 INFO:tasks.ceph.mon.a.smithi017.stderr:2023-09-30T00:49:59.999+0000 7f74e7bdd700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:50:02.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:02.395 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:02.395 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:05.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:05.787 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:05.787 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:08.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:09.179 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:12.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:12.572 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:12.572 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:15.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:15.964 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:15.964 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:18.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:19.360 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:19.360 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:22.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:22.751 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:22.751 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:25.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:29.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:32.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:32.929 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:32.929 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:35.943 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:36.322 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:39.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:39.715 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:39.715 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:42.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:43.109 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:43.109 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:46.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:46.502 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:46.502 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:49.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:49.900 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:49.900 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:52.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:53.297 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:53.297 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:56.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:56.688 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:50:56.688 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:59.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:00.083 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:00.083 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:03.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:03.477 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:03.477 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:06.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:06.871 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:06.871 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:09.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:10.261 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:13.276 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:13.655 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:16.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:17.048 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:17.048 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:20.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:20.448 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:20.448 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:23.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:23.843 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:26.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:30.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:30.629 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:30.630 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:33.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:34.027 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:34.027 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:37.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:40.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:40.817 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:43.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:44.210 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:44.211 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:47.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:47.604 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:47.604 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:50.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:50.997 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:50.997 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:54.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:54.391 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:54.391 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:57.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:57.788 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:51:57.788 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:00.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:01.184 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:01.184 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:04.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:04.578 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:04.579 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:04.592 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:52:04.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:52:04.923 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:04.923 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":18,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:58.979992+0000","last_up_change":"2023-09-30T00:46:53.912335+0000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:46:55.722423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:46:56.041527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":88142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6801","nonce":88142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":88142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":88142}]},"public_addr":"172.21.15.17:6800/88142","cluster_addr":"172.21.15.17:6801/88142","heartbeat_back_addr":"172.21.15.17:6803/88142","heartbeat_front_addr":"172.21.15.17:6802/88142","state":["exists","up"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":88158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6809","nonce":88158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":88158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":88158}]},"public_addr":"172.21.15.17:6808/88158","cluster_addr":"172.21.15.17:6809/88158","heartbeat_back_addr":"172.21.15.17:6811/88158","heartbeat_front_addr":"172.21.15.17:6810/88158","state":["exists","up"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":88143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6805","nonce":88143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":88143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":88143}]},"public_addr":"172.21.15.17:6804/88143","cluster_addr":"172.21.15.17:6805/88143","heartbeat_back_addr":"172.21.15.17:6807/88143","heartbeat_front_addr":"172.21.15.17:6806/88143","state":["exists","up"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":88159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":88159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6815","nonce":88159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":88159}]},"public_addr":"172.21.15.17:6812/88159","cluster_addr":"172.21.15.17:6813/88159","heartbeat_back_addr":"172.21.15.17:6815/88159","heartbeat_front_addr":"172.21.15.17:6814/88159","state":["exists","up"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":88085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6813","nonce":88085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":88085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":88085}]},"public_addr":"172.21.15.71:6812/88085","cluster_addr":"172.21.15.71:6813/88085","heartbeat_back_addr":"172.21.15.71:6815/88085","heartbeat_front_addr":"172.21.15.71:6814/88085","state":["exists","up"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":88070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":88070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":88070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":88070}]},"public_addr":"172.21.15.71:6800/88070","cluster_addr":"172.21.15.71:6801/88070","heartbeat_back_addr":"172.21.15.71:6803/88070","heartbeat_front_addr":"172.21.15.71:6802/88070","state":["exists","up"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":88071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":88071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":88071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":88071}]},"public_addr":"172.21.15.71:6804/88071","cluster_addr":"172.21.15.71:6805/88071","heartbeat_back_addr":"172.21.15.71:6807/88071","heartbeat_front_addr":"172.21.15.71:6806/88071","state":["exists","up"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":88072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":88072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6811","nonce":88072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":88072}]},"public_addr":"172.21.15.71:6808/88072","cluster_addr":"172.21.15.71:6809/88072","heartbeat_back_addr":"172.21.15.71:6811/88072","heartbeat_front_addr":"172.21.15.71:6810/88072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.325066+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.472475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391217+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.428015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.465329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.456368+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.493001+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:52:04.937 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:52:04.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:52:05.261 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:05.262 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:52:05.271 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2023-09-30T00:52:03.730344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002309"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002488+0000","last_change":"2023-09-30T00:46:57.997529+0000","last_active":"2023-09-30T00:46:58.002488+0000","last_peered":"2023-09-30T00:46:58.002488+0000","last_clean":"2023-09-30T00:46:58.002488+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:46:58.002488+0000","last_undegraded":"2023-09-30T00:46:58.002488+0000","last_fullsized":"2023-09-30T00:46:58.002488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:17:07.145653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992796+0000","last_change":"2023-09-30T00:46:57.992123+0000","last_active":"2023-09-30T00:46:57.992796+0000","last_peered":"2023-09-30T00:46:57.992796+0000","last_clean":"2023-09-30T00:46:57.992796+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:46:57.992796+0000","last_undegraded":"2023-09-30T00:46:57.992796+0000","last_fullsized":"2023-09-30T00:46:57.992796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:54:28.140664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.999672+0000","last_change":"2023-09-30T00:46:57.994425+0000","last_active":"2023-09-30T00:46:57.999672+0000","last_peered":"2023-09-30T00:46:57.999672+0000","last_clean":"2023-09-30T00:46:57.999672+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:46:57.999672+0000","last_undegraded":"2023-09-30T00:46:57.999672+0000","last_fullsized":"2023-09-30T00:46:57.999672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:56:54.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002593+0000","last_change":"2023-09-30T00:46:57.997288+0000","last_active":"2023-09-30T00:46:58.002593+0000","last_peered":"2023-09-30T00:46:58.002593+0000","last_clean":"2023-09-30T00:46:58.002593+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:46:58.002593+0000","last_undegraded":"2023-09-30T00:46:58.002593+0000","last_fullsized":"2023-09-30T00:46:58.002593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:15:43.543535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.104195+0000","last_change":"2023-09-30T00:46:56.946070+0000","last_active":"2023-09-30T00:46:58.104195+0000","last_peered":"2023-09-30T00:46:58.104195+0000","last_clean":"2023-09-30T00:46:58.104195+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:46:58.104195+0000","last_undegraded":"2023-09-30T00:46:58.104195+0000","last_fullsized":"2023-09-30T00:46:58.104195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_clean_scrub_stamp":"2023-09-30T00:46:55.928441+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:15.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992211+0000","last_change":"2023-09-30T00:46:57.990797+0000","last_active":"2023-09-30T00:46:57.992211+0000","last_peered":"2023-09-30T00:46:57.992211+0000","last_clean":"2023-09-30T00:46:57.992211+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:46:57.992211+0000","last_undegraded":"2023-09-30T00:46:57.992211+0000","last_fullsized":"2023-09-30T00:46:57.992211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:27:46.015207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.531899+0000","last_change":"2023-09-30T00:46:58.531319+0000","last_active":"2023-09-30T00:46:58.531899+0000","last_peered":"2023-09-30T00:46:58.531899+0000","last_clean":"2023-09-30T00:46:58.531899+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:46:58.531899+0000","last_undegraded":"2023-09-30T00:46:58.531899+0000","last_fullsized":"2023-09-30T00:46:58.531899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.203428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.994043+0000","last_change":"2023-09-30T00:46:57.991533+0000","last_active":"2023-09-30T00:46:57.994043+0000","last_peered":"2023-09-30T00:46:57.994043+0000","last_clean":"2023-09-30T00:46:57.994043+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:46:57.994043+0000","last_undegraded":"2023-09-30T00:46:57.994043+0000","last_fullsized":"2023-09-30T00:46:57.994043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:49:05.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.986609+0000","last_change":"2023-09-30T00:46:57.985988+0000","last_active":"2023-09-30T00:46:57.986609+0000","last_peered":"2023-09-30T00:46:57.986609+0000","last_clean":"2023-09-30T00:46:57.986609+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:46:57.986609+0000","last_undegraded":"2023-09-30T00:46:57.986609+0000","last_fullsized":"2023-09-30T00:46:57.986609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:43:22.827084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.64900000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61599999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.46800000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.316,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.70499999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.40400000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.372,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38600000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.61499999999999999},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56999999999999995},"min":{"1min":0.42799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.40500000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.497},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46700000000000003,"15min":0.48699999999999999},"min":{"1min":0.32600000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.56599999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.36799999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53300000000000003,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76700000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.39000000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57699999999999996,"15min":0.60699999999999998},"min":{"1min":0.38800000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88200000000000001,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.432},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.63500000000000001},"min":{"1min":0.40500000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.82499999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.378,"15min":0.38900000000000001},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.56200000000000006,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40000000000000002,"15min":0.40799999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.52700000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54800000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.32300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63100000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.48899999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.628,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85699999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.45900000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.80000000000000004,"5min":0.877,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83099999999999996,"5min":0.875,"15min":0.875},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81000000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63800000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.88500000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65600000000000003,"15min":0.68100000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.70399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.371,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.29099999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55300000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.52900000000000003,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.438},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.33000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86199999999999999,"5min":1.091,"15min":1.091},"last":0.499},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.874,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.84099999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.53700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88500000000000001,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.52500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.77900000000000003,"5min":1.087,"15min":1.087},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81499999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.51500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.76400000000000001,"5min":1.01,"15min":1.01},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85799999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.442,"15min":0.435},"min":{"1min":0.33100000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42099999999999999,"15min":0.42299999999999999},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39800000000000002},"min":{"1min":0.32200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47099999999999997},"min":{"1min":0.312,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.502}]},{"osd":5,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.44500000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64900000000000002,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65300000000000002,"15min":0.66000000000000003},"min":{"1min":0.5,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.51500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.307},{"interface":"front","average":{"1min":0.374,"5min":0.39100000000000001,"15min":0.39400000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.60199999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.48699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56699999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.44900000000000001},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.376},{"interface":"front","average":{"1min":0.439,"5min":0.48999999999999999,"15min":0.52500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.621,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.63300000000000001},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65100000000000002,"15min":0.69399999999999995},"min":{"1min":0.46600000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.66000000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80300000000000005,"5min":0.879,"15min":0.879},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63300000000000001,"15min":0.66700000000000004},"min":{"1min":0.42799999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.67000000000000004,"15min":0.71399999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58899999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65300000000000002,"15min":0.68899999999999995},"min":{"1min":0.441,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.67400000000000004,"15min":0.70799999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86099999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.628},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.32100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81299999999999994,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54200000000000004,"15min":0.56000000000000005},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54900000000000004,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70399999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.56499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.42799999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.75600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.441,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63200000000000001,"5min":0.877,"15min":0.877},"last":0.316},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.62,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64900000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72399999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51400000000000001,"15min":0.54200000000000004},"min":{"1min":0.36399999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71599999999999997,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68000000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72699999999999998,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53500000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46800000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76100000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73799999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48999999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.505},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72499999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.37}]},{"osd":7,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.753,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.753,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:52:05.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:05.594 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2023-09-30T00:52:03.730344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002309"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002488+0000","last_change":"2023-09-30T00:46:57.997529+0000","last_active":"2023-09-30T00:46:58.002488+0000","last_peered":"2023-09-30T00:46:58.002488+0000","last_clean":"2023-09-30T00:46:58.002488+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:46:58.002488+0000","last_undegraded":"2023-09-30T00:46:58.002488+0000","last_fullsized":"2023-09-30T00:46:58.002488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:17:07.145653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992796+0000","last_change":"2023-09-30T00:46:57.992123+0000","last_active":"2023-09-30T00:46:57.992796+0000","last_peered":"2023-09-30T00:46:57.992796+0000","last_clean":"2023-09-30T00:46:57.992796+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:46:57.992796+0000","last_undegraded":"2023-09-30T00:46:57.992796+0000","last_fullsized":"2023-09-30T00:46:57.992796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:54:28.140664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.999672+0000","last_change":"2023-09-30T00:46:57.994425+0000","last_active":"2023-09-30T00:46:57.999672+0000","last_peered":"2023-09-30T00:46:57.999672+0000","last_clean":"2023-09-30T00:46:57.999672+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:46:57.999672+0000","last_undegraded":"2023-09-30T00:46:57.999672+0000","last_fullsized":"2023-09-30T00:46:57.999672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:56:54.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002593+0000","last_change":"2023-09-30T00:46:57.997288+0000","last_active":"2023-09-30T00:46:58.002593+0000","last_peered":"2023-09-30T00:46:58.002593+0000","last_clean":"2023-09-30T00:46:58.002593+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:46:58.002593+0000","last_undegraded":"2023-09-30T00:46:58.002593+0000","last_fullsized":"2023-09-30T00:46:58.002593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:15:43.543535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.104195+0000","last_change":"2023-09-30T00:46:56.946070+0000","last_active":"2023-09-30T00:46:58.104195+0000","last_peered":"2023-09-30T00:46:58.104195+0000","last_clean":"2023-09-30T00:46:58.104195+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:46:58.104195+0000","last_undegraded":"2023-09-30T00:46:58.104195+0000","last_fullsized":"2023-09-30T00:46:58.104195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_clean_scrub_stamp":"2023-09-30T00:46:55.928441+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:15.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992211+0000","last_change":"2023-09-30T00:46:57.990797+0000","last_active":"2023-09-30T00:46:57.992211+0000","last_peered":"2023-09-30T00:46:57.992211+0000","last_clean":"2023-09-30T00:46:57.992211+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:46:57.992211+0000","last_undegraded":"2023-09-30T00:46:57.992211+0000","last_fullsized":"2023-09-30T00:46:57.992211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:27:46.015207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.531899+0000","last_change":"2023-09-30T00:46:58.531319+0000","last_active":"2023-09-30T00:46:58.531899+0000","last_peered":"2023-09-30T00:46:58.531899+0000","last_clean":"2023-09-30T00:46:58.531899+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:46:58.531899+0000","last_undegraded":"2023-09-30T00:46:58.531899+0000","last_fullsized":"2023-09-30T00:46:58.531899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.203428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.994043+0000","last_change":"2023-09-30T00:46:57.991533+0000","last_active":"2023-09-30T00:46:57.994043+0000","last_peered":"2023-09-30T00:46:57.994043+0000","last_clean":"2023-09-30T00:46:57.994043+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:46:57.994043+0000","last_undegraded":"2023-09-30T00:46:57.994043+0000","last_fullsized":"2023-09-30T00:46:57.994043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:49:05.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.986609+0000","last_change":"2023-09-30T00:46:57.985988+0000","last_active":"2023-09-30T00:46:57.986609+0000","last_peered":"2023-09-30T00:46:57.986609+0000","last_clean":"2023-09-30T00:46:57.986609+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:46:57.986609+0000","last_undegraded":"2023-09-30T00:46:57.986609+0000","last_fullsized":"2023-09-30T00:46:57.986609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:43:22.827084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.64900000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61599999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.46800000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.316,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.70499999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.40400000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.372,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38600000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58999999999999997,"15min":0.61499999999999999},"min":{"1min":0.44,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56999999999999995},"min":{"1min":0.42799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.40500000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.61199999999999999},"min":{"1min":0.41799999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.60199999999999998},"min":{"1min":0.40300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.497},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46700000000000003,"15min":0.48699999999999999},"min":{"1min":0.32600000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.56599999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.36799999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53300000000000003,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76700000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.39000000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Sep 30 00:51:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57699999999999996,"15min":0.60699999999999998},"min":{"1min":0.38800000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.88200000000000001,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.432},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60499999999999998,"15min":0.63500000000000001},"min":{"1min":0.40500000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.82499999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.378,"15min":0.38900000000000001},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.56200000000000006,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40000000000000002,"15min":0.40799999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.52700000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54800000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.32300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63100000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.48899999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.628,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85699999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.45900000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.80000000000000004,"5min":0.877,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83099999999999996,"5min":0.875,"15min":0.875},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81000000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63800000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:51:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.621,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.88500000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65600000000000003,"15min":0.68100000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.70399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.371,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.29099999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55300000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.52900000000000003,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.438},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.33000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86199999999999999,"5min":1.091,"15min":1.091},"last":0.499},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.874,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.84099999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.53700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88500000000000001,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.52500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.77900000000000003,"5min":1.087,"15min":1.087},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81499999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.51500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.76400000000000001,"5min":1.01,"15min":1.01},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85799999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.442,"15min":0.435},"min":{"1min":0.33100000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42099999999999999,"15min":0.42299999999999999},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39800000000000002},"min":{"1min":0.32200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47099999999999997},"min":{"1min":0.312,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.502}]},{"osd":5,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.44500000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64900000000000002,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Sep 30 00:51:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65300000000000002,"15min":0.66000000000000003},"min":{"1min":0.5,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.51500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.307},{"interface":"front","average":{"1min":0.374,"5min":0.39100000000000001,"15min":0.39400000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.49299999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.60199999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.48699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56699999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.44900000000000001},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.376},{"interface":"front","average":{"1min":0.439,"5min":0.48999999999999999,"15min":0.52500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.621,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.63300000000000001},"min":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65100000000000002,"15min":0.69399999999999995},"min":{"1min":0.46600000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.66000000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80300000000000005,"5min":0.879,"15min":0.879},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63300000000000001,"15min":0.66700000000000004},"min":{"1min":0.42799999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.67000000000000004,"15min":0.71399999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58899999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:50:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65300000000000002,"15min":0.68899999999999995},"min":{"1min":0.441,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.67400000000000004,"15min":0.70799999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86099999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.628},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69199999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.32100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81299999999999994,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54200000000000004,"15min":0.56000000000000005},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54900000000000004,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70399999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.56499999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.42799999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.75600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.441,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63200000000000001,"5min":0.877,"15min":0.877},"last":0.316},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.62,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64900000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72399999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Sep 30 00:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51400000000000001,"15min":0.54200000000000004},"min":{"1min":0.36399999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71599999999999997,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68000000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72699999999999998,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53500000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46800000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76100000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51900000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73799999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48999999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42099999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.505},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72499999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.37}]},{"osd":7,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.753,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.753,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:52:05.608 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:52:05.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:52:05.931 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:05.933 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:52:05.944 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2023-09-30T00:52:05.730691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002288"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002488+0000","last_change":"2023-09-30T00:46:57.997529+0000","last_active":"2023-09-30T00:46:58.002488+0000","last_peered":"2023-09-30T00:46:58.002488+0000","last_clean":"2023-09-30T00:46:58.002488+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:46:58.002488+0000","last_undegraded":"2023-09-30T00:46:58.002488+0000","last_fullsized":"2023-09-30T00:46:58.002488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:17:07.145653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992796+0000","last_change":"2023-09-30T00:46:57.992123+0000","last_active":"2023-09-30T00:46:57.992796+0000","last_peered":"2023-09-30T00:46:57.992796+0000","last_clean":"2023-09-30T00:46:57.992796+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:46:57.992796+0000","last_undegraded":"2023-09-30T00:46:57.992796+0000","last_fullsized":"2023-09-30T00:46:57.992796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:54:28.140664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.999672+0000","last_change":"2023-09-30T00:46:57.994425+0000","last_active":"2023-09-30T00:46:57.999672+0000","last_peered":"2023-09-30T00:46:57.999672+0000","last_clean":"2023-09-30T00:46:57.999672+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:46:57.999672+0000","last_undegraded":"2023-09-30T00:46:57.999672+0000","last_fullsized":"2023-09-30T00:46:57.999672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:56:54.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002593+0000","last_change":"2023-09-30T00:46:57.997288+0000","last_active":"2023-09-30T00:46:58.002593+0000","last_peered":"2023-09-30T00:46:58.002593+0000","last_clean":"2023-09-30T00:46:58.002593+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:46:58.002593+0000","last_undegraded":"2023-09-30T00:46:58.002593+0000","last_fullsized":"2023-09-30T00:46:58.002593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:15:43.543535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.104195+0000","last_change":"2023-09-30T00:46:56.946070+0000","last_active":"2023-09-30T00:46:58.104195+0000","last_peered":"2023-09-30T00:46:58.104195+0000","last_clean":"2023-09-30T00:46:58.104195+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:46:58.104195+0000","last_undegraded":"2023-09-30T00:46:58.104195+0000","last_fullsized":"2023-09-30T00:46:58.104195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_clean_scrub_stamp":"2023-09-30T00:46:55.928441+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:15.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992211+0000","last_change":"2023-09-30T00:46:57.990797+0000","last_active":"2023-09-30T00:46:57.992211+0000","last_peered":"2023-09-30T00:46:57.992211+0000","last_clean":"2023-09-30T00:46:57.992211+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:46:57.992211+0000","last_undegraded":"2023-09-30T00:46:57.992211+0000","last_fullsized":"2023-09-30T00:46:57.992211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:27:46.015207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.531899+0000","last_change":"2023-09-30T00:46:58.531319+0000","last_active":"2023-09-30T00:46:58.531899+0000","last_peered":"2023-09-30T00:46:58.531899+0000","last_clean":"2023-09-30T00:46:58.531899+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:46:58.531899+0000","last_undegraded":"2023-09-30T00:46:58.531899+0000","last_fullsized":"2023-09-30T00:46:58.531899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.203428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.994043+0000","last_change":"2023-09-30T00:46:57.991533+0000","last_active":"2023-09-30T00:46:57.994043+0000","last_peered":"2023-09-30T00:46:57.994043+0000","last_clean":"2023-09-30T00:46:57.994043+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:46:57.994043+0000","last_undegraded":"2023-09-30T00:46:57.994043+0000","last_fullsized":"2023-09-30T00:46:57.994043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:49:05.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.986609+0000","last_change":"2023-09-30T00:46:57.985988+0000","last_active":"2023-09-30T00:46:57.986609+0000","last_peered":"2023-09-30T00:46:57.986609+0000","last_clean":"2023-09-30T00:46:57.986609+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:46:57.986609+0000","last_undegraded":"2023-09-30T00:46:57.986609+0000","last_fullsized":"2023-09-30T00:46:57.986609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:43:22.827084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.46800000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.503},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.316,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.70499999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.372,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.53300000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91800000000000004,"5min":0.996,"15min":0.996},"last":0.374}]},{"osd":1,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.32600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72099999999999997,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79100000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56399999999999995,"15min":0.60699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83999999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.39800000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57999999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.03,"15min":1.03},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.73399999999999999,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52000000000000002,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.27300000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68300000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.36099999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60499999999999998},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59199999999999997,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.753,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.372,"15min":0.38700000000000001},"min":{"1min":0.217,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.49399999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.55600000000000005,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52500000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.44600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.315,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.628,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.501},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.48699999999999999},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64400000000000002,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.45200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.67800000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.41999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.81100000000000005,"5min":0.877,"15min":0.877},"last":0.63},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85999999999999999,"5min":0.875,"15min":0.875},"last":0.70999999999999996}]},{"osd":6,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90500000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76300000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.62,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.81000000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.68100000000000005},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.69699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.371,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.29099999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55300000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.52900000000000003,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.438},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.33000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86199999999999999,"5min":1.091,"15min":1.091},"last":0.499},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.874,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.84099999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.53700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88500000000000001,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.52500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.77900000000000003,"5min":1.087,"15min":1.087},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81499999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.51500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.76400000000000001,"5min":1.01,"15min":1.01},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85799999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.38400000000000001},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.55700000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.439,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57899999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.312}]},{"osd":1,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57499999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41299999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53700000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.35099999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64800000000000002,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82799999999999996,"5min":1.107,"15min":1.107},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88100000000000001,"5min":1.137,"15min":1.137},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":1.073,"15min":1.073},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63300000000000001,"15min":0.66400000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81399999999999995,"5min":1.153,"15min":1.153},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65200000000000002,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84799999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66900000000000004,"15min":0.67900000000000005},"min":{"1min":0.47299999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.86099999999999999,"5min":1.123,"15min":1.123},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64400000000000002,"15min":0.66400000000000003},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":1.03,"15min":1.03},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":1.167,"15min":1.167},"last":0.59199999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.224},{"interface":"front","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.39200000000000002},"min":{"1min":0.30499999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.47299999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.434,"15min":0.48099999999999998},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57099999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.375},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46999999999999997,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60199999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.40699999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.629},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.433},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.71099999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.629,"15min":0.68700000000000006},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.79600000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61399999999999999,"15min":0.65500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79200000000000004,"5min":0.879,"15min":0.879},"last":0.45900000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.66100000000000003},"min":{"1min":0.41899999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81599999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.64700000000000002,"15min":0.70699999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95699999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.49399999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63600000000000001,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65200000000000002,"15min":0.69999999999999996},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77900000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.51200000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73099999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55800000000000005},"min":{"1min":0.36899999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71099999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.45900000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71299999999999997,"5min":0.877,"15min":0.877},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67000000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.38}]},{"osd":6,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.503},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.76200000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.58799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68000000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72699999999999998,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.67700000000000005}]},{"osd":1,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76100000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73799999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.505},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72499999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46899999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.753,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.753,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:52:05.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:52:06.271 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:06.272 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":18,"fsid":"68e21822-bcf9-49ad-ac35-975ce04d146e","created":"2023-09-30T00:46:43.884393+0000","modified":"2023-09-30T00:46:58.979992+0000","last_up_change":"2023-09-30T00:46:53.912335+0000","last_in_change":"2023-09-30T00:46:47.228732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:46:55.722423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:46:56.041527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e0fd89fc-05f1-4087-a5e3-1522fb41339b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":88142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6801","nonce":88142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6803","nonce":88142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":88142}]},"public_addr":"172.21.15.17:6800/88142","cluster_addr":"172.21.15.17:6801/88142","heartbeat_back_addr":"172.21.15.17:6803/88142","heartbeat_front_addr":"172.21.15.17:6802/88142","state":["exists","up"]},{"osd":1,"uuid":"320ee357-49bb-4a6b-9676-220ea8d867a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":88158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6809","nonce":88158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6811","nonce":88158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":88158}]},"public_addr":"172.21.15.17:6808/88158","cluster_addr":"172.21.15.17:6809/88158","heartbeat_back_addr":"172.21.15.17:6811/88158","heartbeat_front_addr":"172.21.15.17:6810/88158","state":["exists","up"]},{"osd":2,"uuid":"07c3c896-3c42-4840-8824-ab3e35142c56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":88143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6805","nonce":88143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6807","nonce":88143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":88143}]},"public_addr":"172.21.15.17:6804/88143","cluster_addr":"172.21.15.17:6805/88143","heartbeat_back_addr":"172.21.15.17:6807/88143","heartbeat_front_addr":"172.21.15.17:6806/88143","state":["exists","up"]},{"osd":3,"uuid":"320b1738-9a22-49a6-bb8d-0c5cf165571a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":88159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6813","nonce":88159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6815","nonce":88159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":88159}]},"public_addr":"172.21.15.17:6812/88159","cluster_addr":"172.21.15.17:6813/88159","heartbeat_back_addr":"172.21.15.17:6815/88159","heartbeat_front_addr":"172.21.15.17:6814/88159","state":["exists","up"]},{"osd":4,"uuid":"3a3ae6c2-5973-400f-9d51-0a3a896b225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":88085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6813","nonce":88085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":88085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":88085}]},"public_addr":"172.21.15.71:6812/88085","cluster_addr":"172.21.15.71:6813/88085","heartbeat_back_addr":"172.21.15.71:6815/88085","heartbeat_front_addr":"172.21.15.71:6814/88085","state":["exists","up"]},{"osd":5,"uuid":"a6c34efb-477e-4157-ac55-6f93592ff787","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":88070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6801","nonce":88070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6803","nonce":88070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":88070}]},"public_addr":"172.21.15.71:6800/88070","cluster_addr":"172.21.15.71:6801/88070","heartbeat_back_addr":"172.21.15.71:6803/88070","heartbeat_front_addr":"172.21.15.71:6802/88070","state":["exists","up"]},{"osd":6,"uuid":"8ed0a2c2-6dae-42b5-ab67-a2fac8753605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":88071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":88071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6807","nonce":88071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":88071}]},"public_addr":"172.21.15.71:6804/88071","cluster_addr":"172.21.15.71:6805/88071","heartbeat_back_addr":"172.21.15.71:6807/88071","heartbeat_front_addr":"172.21.15.71:6806/88071","state":["exists","up"]},{"osd":7,"uuid":"a2d20a42-04a9-4102-af3d-5ed71c587ab1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":88072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":88072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6811","nonce":88072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":88072}]},"public_addr":"172.21.15.71:6808/88072","cluster_addr":"172.21.15.71:6809/88072","heartbeat_back_addr":"172.21.15.71:6811/88072","heartbeat_front_addr":"172.21.15.71:6810/88072","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.325066+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.472475+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391217+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.428015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.391149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.465329+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.456368+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:46:52.493001+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:52:07.286 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T00:52:07.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:07.443 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:07.443 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:07.443 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:07.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T00:52:07.782 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T00:52:07.793 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T00:52:07.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:07.937 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:07.937 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:07.937 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:07.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T00:52:08.274 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T00:52:08.285 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T00:52:08.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:08.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T00:52:08.767 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T00:52:08.777 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T00:52:08.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:08.923 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:08.923 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:08.923 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:08.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T00:52:09.261 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T00:52:09.272 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T00:52:09.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:09.417 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:09.417 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:09.417 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:09.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T00:52:09.756 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T00:52:09.767 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T00:52:09.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:09.913 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:09.914 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:09.914 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:09.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T00:52:10.255 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T00:52:10.266 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T00:52:10.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:10.411 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:10.412 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:10.412 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:10.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T00:52:10.749 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T00:52:10.760 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T00:52:10.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:52:10.904 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2023-09-30T00:52:10.904 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:52:10.905 INFO:teuthology.orchestra.run.smithi017.stdout:} 2023-09-30T00:52:10.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T00:52:11.241 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T00:52:11.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:52:11.574 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:11.576 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:52:11.587 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2023-09-30T00:52:09.731319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002199"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002488+0000","last_change":"2023-09-30T00:46:57.997529+0000","last_active":"2023-09-30T00:46:58.002488+0000","last_peered":"2023-09-30T00:46:58.002488+0000","last_clean":"2023-09-30T00:46:58.002488+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:46:58.002488+0000","last_undegraded":"2023-09-30T00:46:58.002488+0000","last_fullsized":"2023-09-30T00:46:58.002488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:17:07.145653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992796+0000","last_change":"2023-09-30T00:46:57.992123+0000","last_active":"2023-09-30T00:46:57.992796+0000","last_peered":"2023-09-30T00:46:57.992796+0000","last_clean":"2023-09-30T00:46:57.992796+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:46:57.992796+0000","last_undegraded":"2023-09-30T00:46:57.992796+0000","last_fullsized":"2023-09-30T00:46:57.992796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:54:28.140664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.999672+0000","last_change":"2023-09-30T00:46:57.994425+0000","last_active":"2023-09-30T00:46:57.999672+0000","last_peered":"2023-09-30T00:46:57.999672+0000","last_clean":"2023-09-30T00:46:57.999672+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:46:57.999672+0000","last_undegraded":"2023-09-30T00:46:57.999672+0000","last_fullsized":"2023-09-30T00:46:57.999672+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:56:54.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.002593+0000","last_change":"2023-09-30T00:46:57.997288+0000","last_active":"2023-09-30T00:46:58.002593+0000","last_peered":"2023-09-30T00:46:58.002593+0000","last_clean":"2023-09-30T00:46:58.002593+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:46:58.002593+0000","last_undegraded":"2023-09-30T00:46:58.002593+0000","last_fullsized":"2023-09-30T00:46:58.002593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:15:43.543535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.104195+0000","last_change":"2023-09-30T00:46:56.946070+0000","last_active":"2023-09-30T00:46:58.104195+0000","last_peered":"2023-09-30T00:46:58.104195+0000","last_clean":"2023-09-30T00:46:58.104195+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:46:58.104195+0000","last_undegraded":"2023-09-30T00:46:58.104195+0000","last_fullsized":"2023-09-30T00:46:58.104195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:55.928441+0000","last_clean_scrub_stamp":"2023-09-30T00:46:55.928441+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:15.055030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.992211+0000","last_change":"2023-09-30T00:46:57.990797+0000","last_active":"2023-09-30T00:46:57.992211+0000","last_peered":"2023-09-30T00:46:57.992211+0000","last_clean":"2023-09-30T00:46:57.992211+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:46:57.992211+0000","last_undegraded":"2023-09-30T00:46:57.992211+0000","last_fullsized":"2023-09-30T00:46:57.992211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:27:46.015207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:58.531899+0000","last_change":"2023-09-30T00:46:58.531319+0000","last_active":"2023-09-30T00:46:58.531899+0000","last_peered":"2023-09-30T00:46:58.531899+0000","last_clean":"2023-09-30T00:46:58.531899+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:46:58.531899+0000","last_undegraded":"2023-09-30T00:46:58.531899+0000","last_fullsized":"2023-09-30T00:46:58.531899+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.203428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.994043+0000","last_change":"2023-09-30T00:46:57.991533+0000","last_active":"2023-09-30T00:46:57.994043+0000","last_peered":"2023-09-30T00:46:57.994043+0000","last_clean":"2023-09-30T00:46:57.994043+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:46:57.994043+0000","last_undegraded":"2023-09-30T00:46:57.994043+0000","last_fullsized":"2023-09-30T00:46:57.994043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:49:05.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:46:57.986609+0000","last_change":"2023-09-30T00:46:57.985988+0000","last_active":"2023-09-30T00:46:57.986609+0000","last_peered":"2023-09-30T00:46:57.986609+0000","last_clean":"2023-09-30T00:46:57.986609+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:46:57.986609+0000","last_undegraded":"2023-09-30T00:46:57.986609+0000","last_fullsized":"2023-09-30T00:46:57.986609+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:46:56.932171+0000","last_clean_scrub_stamp":"2023-09-30T00:46:56.932171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:43:22.827084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.28999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.313,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.46800000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.503},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.438},"min":{"1min":0.316,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.47899999999999998,"15min":0.48599999999999999},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.70499999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.372,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:51:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.752,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.53300000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91800000000000004,"5min":0.996,"15min":0.996},"last":0.374}]},{"osd":1,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.32600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72099999999999997,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79100000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56399999999999995,"15min":0.60699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83999999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.39800000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57999999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.03,"15min":1.03},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.73399999999999999,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52000000000000002,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.27300000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68300000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.36099999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60499999999999998},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59199999999999997,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.753,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.372,"15min":0.38700000000000001},"min":{"1min":0.217,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.49399999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.55600000000000005,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52500000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.44600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.315,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.628,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.501},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.48699999999999999},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64400000000000002,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.45200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.67800000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.41999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.81100000000000005,"5min":0.877,"15min":0.877},"last":0.63},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85999999999999999,"5min":0.875,"15min":0.875},"last":0.70999999999999996}]},{"osd":6,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90500000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76300000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.62,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.81000000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.68100000000000005},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.69699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.371,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.29099999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55300000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39000000000000001,"15min":0.39800000000000002},"min":{"1min":0.29699999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.52900000000000003,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.438},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.33000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.50900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86199999999999999,"5min":1.091,"15min":1.091},"last":0.499},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.874,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.84099999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.53700000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.88500000000000001,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63,"15min":0.63900000000000001},"min":{"1min":0.52500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.77900000000000003,"5min":1.087,"15min":1.087},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64900000000000002,"15min":0.65900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81499999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.51500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.76400000000000001,"5min":1.01,"15min":1.01},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85799999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.38400000000000001},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.55700000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.439,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57899999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.312}]},{"osd":1,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57499999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41299999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53700000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.35099999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64800000000000002,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82799999999999996,"5min":1.107,"15min":1.107},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88100000000000001,"5min":1.137,"15min":1.137},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":1.073,"15min":1.073},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63300000000000001,"15min":0.66400000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81399999999999995,"5min":1.153,"15min":1.153},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65200000000000002,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84799999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66900000000000004,"15min":0.67900000000000005},"min":{"1min":0.47299999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.86099999999999999,"5min":1.123,"15min":1.123},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64400000000000002,"15min":0.66400000000000003},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":1.03,"15min":1.03},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":1.167,"15min":1.167},"last":0.59199999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.224},{"interface":"front","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.39200000000000002},"min":{"1min":0.30499999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.47299999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.434,"15min":0.48099999999999998},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57099999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.375},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46999999999999997,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60199999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.40699999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.629},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.433},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.71099999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.629,"15min":0.68700000000000006},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.79600000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61399999999999999,"15min":0.65500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79200000000000004,"5min":0.879,"15min":0.879},"last":0.45900000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.66100000000000003},"min":{"1min":0.41899999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81599999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.64700000000000002,"15min":0.70699999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95699999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.49399999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63600000000000001,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65200000000000002,"15min":0.69999999999999996},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77900000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.51200000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73099999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55800000000000005},"min":{"1min":0.36899999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71099999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.45900000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71299999999999997,"5min":0.877,"15min":0.877},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67000000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.38}]},{"osd":6,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.503},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.76200000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.58799999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68000000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72699999999999998,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.67700000000000005}]},{"osd":1,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.41399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76100000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.747,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57699999999999996,"15min":0.57799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73799999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68899999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.505},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72499999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46899999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:51:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.753,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.753,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:52:11.596 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.597 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.597 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.597 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.597 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T00:46:55.928441+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.597 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.598 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.598 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.598 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T00:46:56.932171+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=46, tm_sec=56, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=52, tm_sec=6, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:52:11.598 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T00:52:31.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:52:31.938 INFO:teuthology.orchestra.run.smithi017.stdout: 2023-09-30T00:52:31.939 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2023-09-30T00:52:31.950 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2023-09-30T00:52:31.736159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":42148,"kb_used_data":1252,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":754932572,"statfs":{"total":773094113280,"available":773050953728,"internally_reserved":0,"allocated":1282048,"data_stored":1440782,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:11.092821+0000","last_change":"2023-09-30T00:52:11.092783+0000","last_active":"2023-09-30T00:52:11.092821+0000","last_peered":"2023-09-30T00:52:11.092821+0000","last_clean":"2023-09-30T00:52:11.092821+0000","last_became_active":"2023-09-30T00:46:57.997279+0000","last_became_peered":"2023-09-30T00:46:57.997279+0000","last_unstale":"2023-09-30T00:52:11.092821+0000","last_undegraded":"2023-09-30T00:52:11.092821+0000","last_fullsized":"2023-09-30T00:52:11.092821+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:11.092678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:11.092678+0000","last_clean_scrub_stamp":"2023-09-30T00:52:11.092678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:21:19.756031+0000","scrub_duration":0.001653084,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:09.755976+0000","last_change":"2023-09-30T00:52:09.755952+0000","last_active":"2023-09-30T00:52:09.755976+0000","last_peered":"2023-09-30T00:52:09.755976+0000","last_clean":"2023-09-30T00:52:09.755976+0000","last_became_active":"2023-09-30T00:46:57.991821+0000","last_became_peered":"2023-09-30T00:46:57.991821+0000","last_unstale":"2023-09-30T00:52:09.755976+0000","last_undegraded":"2023-09-30T00:52:09.755976+0000","last_fullsized":"2023-09-30T00:52:09.755976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:09.755871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:09.755871+0000","last_clean_scrub_stamp":"2023-09-30T00:52:09.755871+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:22:14.830213+0000","scrub_duration":0.0016090830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:27.226116+0000","last_change":"2023-09-30T00:52:27.226090+0000","last_active":"2023-09-30T00:52:27.226116+0000","last_peered":"2023-09-30T00:52:27.226116+0000","last_clean":"2023-09-30T00:52:27.226116+0000","last_became_active":"2023-09-30T00:46:57.994144+0000","last_became_peered":"2023-09-30T00:46:57.994144+0000","last_unstale":"2023-09-30T00:52:27.226116+0000","last_undegraded":"2023-09-30T00:52:27.226116+0000","last_fullsized":"2023-09-30T00:52:27.226116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:27.225988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:27.225988+0000","last_clean_scrub_stamp":"2023-09-30T00:52:27.225988+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:29.812390+0000","scrub_duration":0.0018304250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:20.253116+0000","last_change":"2023-09-30T00:52:20.253087+0000","last_active":"2023-09-30T00:52:20.253116+0000","last_peered":"2023-09-30T00:52:20.253116+0000","last_clean":"2023-09-30T00:52:20.253116+0000","last_became_active":"2023-09-30T00:46:57.997028+0000","last_became_peered":"2023-09-30T00:46:57.997028+0000","last_unstale":"2023-09-30T00:52:20.253116+0000","last_undegraded":"2023-09-30T00:52:20.253116+0000","last_fullsized":"2023-09-30T00:52:20.253116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:20.252986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:20.252986+0000","last_clean_scrub_stamp":"2023-09-30T00:52:20.252986+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:44:57.693308+0000","scrub_duration":0.0019041419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'119","reported_seq":173,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:14.311174+0000","last_change":"2023-09-30T00:52:14.311137+0000","last_active":"2023-09-30T00:52:14.311174+0000","last_peered":"2023-09-30T00:52:14.311174+0000","last_clean":"2023-09-30T00:52:14.311174+0000","last_became_active":"2023-09-30T00:46:56.945716+0000","last_became_peered":"2023-09-30T00:46:56.945716+0000","last_unstale":"2023-09-30T00:52:14.311174+0000","last_undegraded":"2023-09-30T00:52:14.311174+0000","last_fullsized":"2023-09-30T00:52:14.311174+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'119","last_scrub_stamp":"2023-09-30T00:52:14.311042+0000","last_deep_scrub":"18'119","last_deep_scrub_stamp":"2023-09-30T00:52:14.311042+0000","last_clean_scrub_stamp":"2023-09-30T00:52:14.311042+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:58:09.707710+0000","scrub_duration":0.0045786769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:13.059541+0000","last_change":"2023-09-30T00:52:13.059497+0000","last_active":"2023-09-30T00:52:13.059541+0000","last_peered":"2023-09-30T00:52:13.059541+0000","last_clean":"2023-09-30T00:52:13.059541+0000","last_became_active":"2023-09-30T00:46:57.990604+0000","last_became_peered":"2023-09-30T00:46:57.990604+0000","last_unstale":"2023-09-30T00:52:13.059541+0000","last_undegraded":"2023-09-30T00:52:13.059541+0000","last_fullsized":"2023-09-30T00:52:13.059541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:13.059368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:13.059368+0000","last_clean_scrub_stamp":"2023-09-30T00:52:13.059368+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:47.200563+0000","scrub_duration":0.0018901530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:13.416159+0000","last_change":"2023-09-30T00:52:13.416133+0000","last_active":"2023-09-30T00:52:13.416159+0000","last_peered":"2023-09-30T00:52:13.416159+0000","last_clean":"2023-09-30T00:52:13.416159+0000","last_became_active":"2023-09-30T00:46:58.530996+0000","last_became_peered":"2023-09-30T00:46:58.530996+0000","last_unstale":"2023-09-30T00:52:13.416159+0000","last_undegraded":"2023-09-30T00:52:13.416159+0000","last_fullsized":"2023-09-30T00:52:13.416159+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:13.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:13.416033+0000","last_clean_scrub_stamp":"2023-09-30T00:52:13.416033+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:06.687565+0000","scrub_duration":0.001692827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:12.101666+0000","last_change":"2023-09-30T00:52:12.101640+0000","last_active":"2023-09-30T00:52:12.101666+0000","last_peered":"2023-09-30T00:52:12.101666+0000","last_clean":"2023-09-30T00:52:12.101666+0000","last_became_active":"2023-09-30T00:46:57.991321+0000","last_became_peered":"2023-09-30T00:46:57.991321+0000","last_unstale":"2023-09-30T00:52:12.101666+0000","last_undegraded":"2023-09-30T00:52:12.101666+0000","last_fullsized":"2023-09-30T00:52:12.101666+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:12.101552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:12.101552+0000","last_clean_scrub_stamp":"2023-09-30T00:52:12.101552+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:59.604305+0000","scrub_duration":0.0018874600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:52:08.800471+0000","last_change":"2023-09-30T00:52:08.800437+0000","last_active":"2023-09-30T00:52:08.800471+0000","last_peered":"2023-09-30T00:52:08.800471+0000","last_clean":"2023-09-30T00:52:08.800471+0000","last_became_active":"2023-09-30T00:46:57.985694+0000","last_became_peered":"2023-09-30T00:46:57.985694+0000","last_unstale":"2023-09-30T00:52:08.800471+0000","last_undegraded":"2023-09-30T00:52:08.800471+0000","last_fullsized":"2023-09-30T00:52:08.800471+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:52:08.800284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:52:08.800284+0000","last_clean_scrub_stamp":"2023-09-30T00:52:08.800284+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:20.391596+0000","scrub_duration":0.001449217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":16592,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5648,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366192,"statfs":{"total":96636764160,"available":96630980608,"internally_reserved":0,"allocated":204800,"data_stored":577279,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.32900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68400000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.65000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56699999999999995,"15min":0.56399999999999995},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.755,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59199999999999997,"15min":0.59699999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73399999999999999,"5min":0.872,"15min":0.872},"last":0.71399999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53900000000000003,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.42299999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.68100000000000005}]},{"osd":3,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.42299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69499999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.437,"15min":0.435},"min":{"1min":0.29599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51400000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.48399999999999999},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.67800000000000005,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48599999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.73599999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36299999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.745,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.73799999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.41999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71099999999999997,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.70499999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":47958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.53300000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.61099999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91800000000000004,"5min":0.996,"15min":0.996},"last":0.73399999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.32600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72099999999999997,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82499999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70099999999999996}]},{"osd":2,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79100000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56399999999999995,"15min":0.60699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83999999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57999999999999996,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.03,"15min":1.03},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.39200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.73399999999999999,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64600000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.48399999999999999},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52000000000000002,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71099999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68300000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.60499999999999998},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59199999999999997,"15min":0.63100000000000001},"min":{"1min":0.45000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.753,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.51400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5148,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366692,"statfs":{"total":96636764160,"available":96631492608,"internally_reserved":0,"allocated":151552,"data_stored":48351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.372,"15min":0.38700000000000001},"min":{"1min":0.217,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.49399999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.40400000000000003,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.59299999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.55600000000000005,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52500000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.31}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.315,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.628,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.48699999999999999},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64400000000000002,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.373}]},{"osd":4,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.45200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.61799999999999999},"min":{"1min":0.41999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.81100000000000005,"5min":0.877,"15min":0.877},"last":0.499},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85999999999999999,"5min":0.875,"15min":0.875},"last":0.39700000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.621,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90500000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.47299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.62,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.81000000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.68100000000000005},"min":{"1min":0.46200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.54500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":576705,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.36399999999999999,"15min":0.373},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.312},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.26000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.64100000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.44900000000000001,"15min":0.45500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64600000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.34499999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27500000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59699999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.441,"15min":0.45100000000000001},"min":{"1min":0.31,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57199999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89300000000000002,"5min":1.091,"15min":1.091},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.46800000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63400000000000001,"15min":0.64600000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86599999999999999,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.45400000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.84899999999999998,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.63600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83199999999999996,"5min":1.087,"15min":1.087},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64300000000000002,"15min":0.65700000000000003},"min":{"1min":0.52400000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.82699999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.496}]},{"osd":7,"last update":"Sat Sep 30 00:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79400000000000004,"5min":1.01,"15min":1.01},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.67100000000000004},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88700000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5124,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366716,"statfs":{"total":96636764160,"available":96631517184,"internally_reserved":0,"allocated":126976,"data_stored":45142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.38400000000000001},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.55700000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.439,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57899999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57499999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41299999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53700000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58299999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64800000000000002,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82799999999999996,"5min":1.107,"15min":1.107},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88100000000000001,"5min":1.137,"15min":1.137},"last":1}]},{"osd":5,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":1.073,"15min":1.073},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63300000000000001,"15min":0.66400000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81399999999999995,"5min":1.153,"15min":1.153},"last":0.96299999999999997}]},{"osd":6,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65200000000000002,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84799999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66900000000000004,"15min":0.67900000000000005},"min":{"1min":0.47299999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.86099999999999999,"5min":1.123,"15min":1.123},"last":1.016}]},{"osd":7,"last update":"Sat Sep 30 00:52:04 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64400000000000002,"15min":0.66400000000000003},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":1.03,"15min":1.03},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.48699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":1.167,"15min":1.167},"last":0.76200000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":46297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.39200000000000002},"min":{"1min":0.30499999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.47299999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.372},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.434,"15min":0.48099999999999998},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60299999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57099999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46999999999999997,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60199999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.629},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.71099999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.629,"15min":0.68700000000000006},"min":{"1min":0.45700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.79600000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61399999999999999,"15min":0.65500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79200000000000004,"5min":0.879,"15min":0.879},"last":0.52100000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.66100000000000003},"min":{"1min":0.41899999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81599999999999995,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.64700000000000002,"15min":0.70699999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95699999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Sep 30 00:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63600000000000001,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65200000000000002,"15min":0.69999999999999996},"min":{"1min":0.49199999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.77900000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.748}]}]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73999999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52700000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73099999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55800000000000005},"min":{"1min":0.36899999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54500000000000004,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71099999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56899999999999995,"15min":0.58999999999999997},"min":{"1min":0.45900000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.76600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71299999999999997,"5min":0.877,"15min":0.877},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67000000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.503},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54500000000000004,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.622}]},{"osd":7,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.50600000000000001,"15min":0.54000000000000004},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.76200000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.58399999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.65500000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56000000000000005}]},{"osd":1,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70699999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61599999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56000000000000005,"15min":0.54800000000000004},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72799999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57099999999999995,"15min":0.57599999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57699999999999996,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67600000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63100000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45900000000000002,"15min":0.48799999999999999},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59799999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.499,"15min":0.51700000000000002},"min":{"1min":0.29799999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68799999999999994,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67400000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.37},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.51000000000000001,"15min":0.52000000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60799999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41199999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54800000000000004,"15min":0.56599999999999995},"min":{"1min":0.38200000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64800000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.44},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.59299999999999997,"15min":0.60499999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70199999999999996,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":8296,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:52:31.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:52:32.293 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:52:32.295 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:52:32.295 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:52:32.295 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:52:32.295 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.333 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:52:32.334 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:52:32.334 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.398 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:52:32.398 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:52:32.398 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.462 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:52:32.462 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:52:32.462 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.531 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:52:32.531 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T00:52:32.532 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.568 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T00:52:32.568 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T00:52:32.568 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.645 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T00:52:32.646 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T00:52:32.646 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.713 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T00:52:32.714 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T00:52:32.714 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.786 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T00:52:32.786 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:52:32.786 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T00:52:32.786 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.812 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T00:52:32.813 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:52:32.813 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.852 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:52:32.852 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:52:32.852 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:52:32.852 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.863 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:52:32.863 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T00:52:32.863 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.913 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T00:52:32.913 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T00:52:32.913 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-09-30T00:52:32.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-30T00:52:32.926 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T00:52:32.926 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:52:32.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:52:32.983 INFO:teuthology.orchestra.run.smithi017.stdout:1696034809.69229 mgr.y (mgr.4101) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:52:32.984 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:52:32.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:52:33.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:52:33.123 INFO:teuthology.orchestra.run.smithi017.stdout:1696034809.69229 mgr.y (mgr.4101) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:52:33.123 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:52:33.123 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:52:33.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:52:33.309 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:52:33.379 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:52:33.379 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:52:33.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:52:33.481 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:52:33.542 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:52:33.542 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T00:52:33.690 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:52:33.691 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T00:52:33.766 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:52:33.767 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T00:52:33.834 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:52:33.835 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T00:52:33.898 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:52:33.899 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/data/mon.a.tgz 2023-09-30T00:52:33.900 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2023-09-30T00:52:33.915 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.qk4hQmyzUh 2023-09-30T00:52:33.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qk4hQmyzUh 2023-09-30T00:52:34.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.qk4hQmyzUh 2023-09-30T00:52:34.124 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.qk4hQmyzUh is 252KB 2023-09-30T00:52:34.189 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.qk4hQmyzUh 2023-09-30T00:52:34.204 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/data/mon.c.tgz 2023-09-30T00:52:34.204 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2023-09-30T00:52:34.258 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.UOfi9coT6J 2023-09-30T00:52:34.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.UOfi9coT6J 2023-09-30T00:52:34.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.UOfi9coT6J 2023-09-30T00:52:34.473 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.UOfi9coT6J is 343KB 2023-09-30T00:52:34.572 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.UOfi9coT6J 2023-09-30T00:52:34.586 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/data/mon.b.tgz 2023-09-30T00:52:34.587 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.CPgD7yBCwl 2023-09-30T00:52:34.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.CPgD7yBCwl 2023-09-30T00:52:34.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.CPgD7yBCwl 2023-09-30T00:52:34.822 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.CPgD7yBCwl is 343KB 2023-09-30T00:52:34.917 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.CPgD7yBCwl 2023-09-30T00:52:34.931 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:52:34.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:52:34.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:52:34.995 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:52:34.995 DEBUG:teuthology.orchestra.run.smithi017:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:52:35.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:52:35.025 INFO:teuthology.orchestra.run.smithi017.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:52:35.060 INFO:teuthology.orchestra.run.smithi071.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:52:35.066 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:52:35.067 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/remote/smithi017/crash 2023-09-30T00:52:35.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:52:35.099 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/remote/smithi071/crash 2023-09-30T00:52:35.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:52:35.129 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:52:35.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:52:35.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:52:36.073 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:52:36.073 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/remote/smithi017/log 2023-09-30T00:52:36.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:52:36.337 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/remote/smithi071/log 2023-09-30T00:52:36.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:52:36.440 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:52:36.508 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7c01558a59f643df87af2fb2ffe8c77b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:52:36.511 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:52:36.530 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:52:36.558 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:52:36.559 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:52:36.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:52:36.651 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:52:36.661 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:52:36.662 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:52:36.664 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:52:36.676 INFO:teuthology.orchestra.run.smithi017.stderr:bash: ntpq: command not found 2023-09-30T00:52:36.678 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2023-09-30T00:52:36.679 INFO:teuthology.orchestra.run.smithi017.stdout:210 Number of sources = 4 2023-09-30T00:52:36.680 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:52:36.680 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2023-09-30T00:52:36.680 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 -292us[ -280us] +/- 89ms 2023-09-30T00:52:36.680 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -348us[ -336us] +/- 54ms 2023-09-30T00:52:36.680 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +752us[ +764us] +/- 74ms 2023-09-30T00:52:36.680 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:52:36.681 INFO:teuthology.orchestra.run.smithi071.stdout:210 Number of sources = 4 2023-09-30T00:52:36.681 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:52:36.682 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-09-30T00:52:36.682 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -281us[ -260us] +/- 89ms 2023-09-30T00:52:36.682 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -342us[ -320us] +/- 54ms 2023-09-30T00:52:36.682 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +754us[ +775us] +/- 74ms 2023-09-30T00:52:36.682 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:52:36.682 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:52:36.691 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:52:36.691 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:52:36.701 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:52:36.813 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:52:36.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:52:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:52:36.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:52:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:52:36.963 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:52:36.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:52:36.982 INFO:teuthology.task.internal:Duration was 1191.725036 seconds 2023-09-30T00:52:36.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:52:36.993 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:52:36.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:52:36.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:52:37.040 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:52:37.046 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:52:37.250 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:52:37.250 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2023-09-30T00:52:37.251 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:52:37.270 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2023-09-30T00:52:37.271 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:52:37.291 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:52:37.291 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:52:37.313 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:52:37.369 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:52:37.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:52:37.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:52:37.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:52:37.489 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:52:37.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:52:37.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:52:37.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:52:37.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:52:37.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:52:37.580 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2023-09-30T00:52:37.604 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2023-09-30T00:52:37.617 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:52:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:52:37.648 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:52:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:52:37.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:52:37.681 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:52:37.682 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/remote/smithi017 2023-09-30T00:52:37.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:52:37.726 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764/remote/smithi071 2023-09-30T00:52:37.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:52:37.761 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:52:37.761 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:52:37.764 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:52:37.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:52:37.821 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:52:37.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:52:37.833 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:52:37.833 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:52:37.836 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:52:37.849 INFO:teuthology.orchestra.run.smithi017.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:52 /home/ubuntu/cephtest 2023-09-30T00:52:37.864 INFO:teuthology.orchestra.run.smithi071.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:52 /home/ubuntu/cephtest 2023-09-30T00:52:37.865 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:52:37.875 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:52:37.899 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:52:37.924 DEBUG:teuthology.nuke:shortname: smithi017 2023-09-30T00:52:37.925 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:52:37.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-30T00:52:37.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-09-30T00:52:37.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-09-30T00:52:37.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-09-30T00:52:37.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-09-30T00:52:37.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-09-30T00:52:37.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-09-30T00:52:37.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-09-30T00:52:37.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-30T00:52:37.953 DEBUG:teuthology.nuke:shortname: smithi071 2023-09-30T00:52:37.954 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:52:37.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:25:21.443005', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZlaQEP+7adkueMb/+ERFQy3Dwhho1Wb12ZhXccY71Dnj1nVL4Fj5qVr8Ipasufq3xp01VCo5NHXPqmMBXL7Sc='} 2023-09-30T00:52:37.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406764', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:25:21.444559', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJJ2MZTCXYCd/ZJ7xqPR+owGRouh0wJqcJ14wyaj2b535kQMDTxSikwVVcIMqPOdR3lKn5wcwb3bg42m1dINiU='} 2023-09-30T00:52:38.010 INFO:teuthology.orchestra.console.smithi017:Power off 2023-09-30T00:52:38.011 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:52:38.034 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2023-09-30T00:52:38.035 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:52:38.049 INFO:teuthology.orchestra.console.smithi071:Power off 2023-09-30T00:52:38.050 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:52:38.059 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2023-09-30T00:52:38.072 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2023-09-30T00:52:38.072 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:52:38.099 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-09-30T00:52:42.061 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:52:42.101 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:52:42.171 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2023-09-30T00:52:42.211 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-09-30T00:52:46.174 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:52:46.212 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:52:46.284 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2023-09-30T00:52:46.322 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2023-09-30T00:52:46.385 INFO:teuthology.orchestra.console.smithi017:Power off completed 2023-09-30T00:52:46.424 INFO:teuthology.orchestra.console.smithi071:Power off completed 2023-09-30T00:52:46.529 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} duration: 1191.7250363826752 failure_reason: '"1696034809.69229 mgr.y (mgr.4101) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.y: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7c01558a59f643df87af2fb2ffe8c77b status: fail success: false 2023-09-30T00:52:46.530 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:52:46.624 INFO:teuthology.run:FAIL