2023-09-30T02:15:00.225 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T02:15:00.225 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271 --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/orig.config.yaml 2023-09-30T02:15:00.273 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T02:15:00.359 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} email: yweinste@redhat.com first_in_suite: false job_id: '7407271' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon election default strategy: 1 ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd max scrubs: 3 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - must scrub before tier agent can activate sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYOankZWKJky5kfFILmXO0eIT66frOaGpGYvbpK0S+mqX8fpoL6QLneZhLl/VEuLdRf2xx07HivIisYwtRluaw= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF8TKoW78maGXgk+0aMXgSg1b+cRx/KzgQ1eHyyhdluTQs7ZFD21iieBGxAfqEz43v1NGebJS9xtuog7nnLnJ/g= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - sudo ceph osd pool set cache min_write_recency_for_promote 2 - rados: clients: - client.0 objects: 500 op_weights: copy_from: 50 delete: 50 read: 100 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T02:15:00.360 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T02:15:00.361 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T02:15:00.361 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T02:15:00.362 INFO:teuthology.task.internal:Checking packages... 2023-09-30T02:15:00.385 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T02:15:00.385 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T02:15:00.385 INFO:teuthology.packaging:ref: None 2023-09-30T02:15:00.386 INFO:teuthology.packaging:tag: None 2023-09-30T02:15:00.386 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T02:15:00.386 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:15:00.386 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T02:15:00.533 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T02:15:00.533 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T02:15:00.542 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T02:15:00.543 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T02:15:00.562 INFO:teuthology.task.internal:Saving configuration 2023-09-30T02:15:00.579 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T02:15:00.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:15:00.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 02:09:32.052576', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYOankZWKJky5kfFILmXO0eIT66frOaGpGYvbpK0S+mqX8fpoL6QLneZhLl/VEuLdRf2xx07HivIisYwtRluaw='} 2023-09-30T02:15:00.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 02:09:32.051168', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF8TKoW78maGXgk+0aMXgSg1b+cRx/KzgQ1eHyyhdluTQs7ZFD21iieBGxAfqEz43v1NGebJS9xtuog7nnLnJ/g='} 2023-09-30T02:15:00.639 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T02:15:00.650 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T02:15:00.650 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T02:15:00.651 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T02:15:00.733 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb3c59e3ee0>, signals=[15]) 2023-09-30T02:15:00.734 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T02:15:00.747 INFO:teuthology.task.internal:Opening connections... 2023-09-30T02:15:00.748 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:15:00.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:15:00.819 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:15:00.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:15:00.889 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T02:15:00.898 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2023-09-30T02:15:00.921 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2023-09-30T02:15:00.921 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T02:15:00.982 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2023-09-30T02:15:00.983 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T02:15:00.983 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T02:15:00.983 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T02:15:00.983 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2023-09-30T02:15:01.008 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2023-09-30T02:15:01.030 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2023-09-30T02:15:01.030 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2023-09-30T02:15:01.090 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2023-09-30T02:15:01.090 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8" 2023-09-30T02:15:01.090 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T02:15:01.091 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T02:15:01.092 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2023-09-30T02:15:01.116 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T02:15:01.128 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T02:15:01.137 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T02:15:01.137 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2023-09-30T02:15:01.139 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2023-09-30T02:15:01.160 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T02:15:01.169 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T02:15:01.169 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2023-09-30T02:15:01.202 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2023-09-30T02:15:01.228 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T02:15:01.288 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T02:15:01.305 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T02:15:01.305 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T02:15:01.306 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T02:15:01.306 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T02:15:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:15:01.327 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-09-30T02:15:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:15:01.329 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-09-30T02:15:01.387 INFO:teuthology.orchestra.run.smithi157.stdout:4.18.0-489.el8.x86_64 2023-09-30T02:15:01.388 INFO:teuthology.task.kernel:Running kernel on smithi157: 4.18.0-489.el8.x86_64 2023-09-30T02:15:01.388 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2023-09-30T02:15:01.389 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-489.el8.x86_64 2023-09-30T02:15:01.389 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-489.el8.x86_64 2023-09-30T02:15:01.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2023-09-30T02:15:05.168 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - AppStream 28 MB/s | 33 MB 00:01 2023-09-30T02:15:06.397 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 14 MB/s | 33 MB 00:02 2023-09-30T02:15:13.852 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 48 MB 00:02 2023-09-30T02:15:14.198 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 27 MB/s | 48 MB 00:01 2023-09-30T02:15:20.926 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras 73 kB/s | 18 kB 00:00 2023-09-30T02:15:21.329 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2023-09-30T02:15:21.510 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T02:15:21.820 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-09-30T02:15:22.029 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-30T02:15:22.356 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-09-30T02:15:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-09-30T02:15:23.823 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2023-09-30T02:15:27.250 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 612 kB/s | 24 kB 00:00 2023-09-30T02:15:27.629 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 631 kB/s | 24 kB 00:00 2023-09-30T02:15:30.766 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T02:15:30.766 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T02:15:30.999 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:15:31.000 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:15:31.000 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-09-30T02:15:31.000 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:15:31.000 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:15:31.000 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout:Install 3 Packages 2023-09-30T02:15:31.001 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:15:31.002 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 88 M 2023-09-30T02:15:31.002 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 96 M 2023-09-30T02:15:31.002 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:15:31.112 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T02:15:31.112 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T02:15:31.384 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:15:31.384 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:15:31.385 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:15:31.386 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:15:31.386 INFO:teuthology.orchestra.run.smithi157.stdout:Install 3 Packages 2023-09-30T02:15:31.386 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:15:31.386 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 88 M 2023-09-30T02:15:31.386 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 96 M 2023-09-30T02:15:31.386 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:15:31.668 INFO:teuthology.orchestra.run.smithi115.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2023-09-30T02:15:32.264 INFO:teuthology.orchestra.run.smithi157.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2023-09-30T02:15:32.326 INFO:teuthology.orchestra.run.smithi115.stdout:(2/3): kernel-core-4.18.0-514.el8.x86_64.rpm 33 MB/s | 43 MB 00:01 2023-09-30T02:15:32.981 INFO:teuthology.orchestra.run.smithi157.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 25 MB/s | 35 MB 00:01 2023-09-30T02:15:33.151 INFO:teuthology.orchestra.run.smithi115.stdout:(3/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 17 MB/s | 35 MB 00:02 2023-09-30T02:15:33.152 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:15:33.152 INFO:teuthology.orchestra.run.smithi115.stdout:Total 41 MB/s | 88 MB 00:02 2023-09-30T02:15:33.514 INFO:teuthology.orchestra.run.smithi157.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 22 MB/s | 43 MB 00:01 2023-09-30T02:15:33.515 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:15:33.515 INFO:teuthology.orchestra.run.smithi157.stdout:Total 41 MB/s | 88 MB 00:02 2023-09-30T02:15:33.819 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:15:34.187 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:15:34.239 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:15:34.239 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:15:34.606 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:15:34.606 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:15:35.469 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:15:35.496 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:15:35.830 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:15:35.850 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:15:38.245 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:15:38.605 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:15:38.789 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T02:15:39.118 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T02:15:42.394 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T02:15:42.751 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T02:15:42.846 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T02:15:43.159 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T02:15:46.225 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T02:15:46.526 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:15:46.540 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T02:15:46.872 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.284 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.307 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.537 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.561 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.643 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T02:16:08.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T02:16:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T02:16:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T02:16:09.394 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:09.395 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:16:09.395 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:16:09.395 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T02:16:09.395 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T02:16:09.395 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:16:09.395 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:16:09.565 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T02:16:09.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T02:16:09.673 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:16:09.673 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:16:09.673 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T02:16:09.673 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T02:16:09.673 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:16:09.673 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:16:09.779 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T02:16:09.780 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T02:16:09.780 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T02:16:09.780 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T02:16:09.780 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T02:16:09.780 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T02:16:09.780 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T02:16:09.780 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T02:16:09.820 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T02:16:09.988 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T02:16:09.988 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T02:16:09.988 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.02 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 129.el8 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T02:16:10.022 INFO:teuthology.orchestra.run.smithi115.stdout:Group : System Environment/Base 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 9265586 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Relocations : (not relocatable) 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : CentOS BuildSystem 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2023-09-30T02:16:10.023 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:16:10.024 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T02:16:10.024 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T02:16:10.024 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T02:16:10.024 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2023-09-30T02:16:10.024 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:16:10.024 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T02:16:10.025 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T02:16:10.025 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 4.18.0-514.el8.x86_64 2023-09-30T02:16:10.025 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T02:16:10.038 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T02:16:10.038 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T02:16:10.038 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T02:16:10.038 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T02:16:10.038 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T02:16:10.038 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T02:16:10.038 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T02:16:10.039 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T02:16:10.247 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T02:16:10.247 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T02:16:10.247 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2023-09-30T02:16:10.280 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2023-09-30T02:16:10.280 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2023-09-30T02:16:10.280 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.02 2023-09-30T02:16:10.280 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 129.el8 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Group : System Environment/Base 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 9265586 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Relocations : (not relocatable) 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : CentOS BuildSystem 2023-09-30T02:16:10.281 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:16:10.282 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T02:16:10.283 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T02:16:10.283 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 4.18.0-514.el8.x86_64 2023-09-30T02:16:10.283 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T02:16:11.857 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2023-09-30T02:16:12.028 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2023-09-30T02:16:14.132 INFO:teuthology.orchestra.run.smithi115.stderr:done 2023-09-30T02:16:14.134 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-09-30T02:16:14.157 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.SBqvBBdoO8 2023-09-30T02:16:14.158 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SBqvBBdoO8 2023-09-30T02:16:14.255 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.SBqvBBdoO8 2023-09-30T02:16:14.301 INFO:teuthology.orchestra.run.smithi157.stderr:done 2023-09-30T02:16:14.303 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-09-30T02:16:14.345 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.4zvUckf9Rk 2023-09-30T02:16:14.345 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4zvUckf9Rk 2023-09-30T02:16:14.356 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.SBqvBBdoO8 is 6KB 2023-09-30T02:16:14.404 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.SBqvBBdoO8 2023-09-30T02:16:14.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.4zvUckf9Rk 2023-09-30T02:16:14.421 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T02:16:14.495 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T02:16:14.495 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T02:16:14.495 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T02:16:14.495 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T02:16:14.496 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T02:16:14.581 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.4zvUckf9Rk is 6KB 2023-09-30T02:16:14.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2023-09-30T02:16:14.630 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.4zvUckf9Rk 2023-09-30T02:16:14.647 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T02:16:14.725 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T02:16:14.725 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T02:16:14.725 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T02:16:14.726 INFO:teuthology.orchestra.run.smithi157.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T02:16:14.726 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T02:16:14.875 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2023-09-30T02:16:44.660 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T02:16:44.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T02:16:44.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:16:44.878 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T02:16:44.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:16:44.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:17:19.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-09-30T02:17:19.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T02:17:28.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T02:17:28.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:17:28.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:17:28.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:17:31.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T02:17:43.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:17:43.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:17:46.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-09-30T02:17:58.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T02:17:58.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:17:58.999 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-09-30T02:17:59.266 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T02:17:59.266 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-30T02:17:59.267 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-09-30T02:17:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-514.el8.x86_64 2023-09-30T02:17:59.324 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T02:17:59.324 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T02:17:59.324 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T02:18:00.325 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T02:18:00.326 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T02:18:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2023-09-30T02:18:00.405 DEBUG:teuthology.parallel:result is None 2023-09-30T02:18:01.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T02:18:16.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:18:16.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:18:17.402 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-09-30T02:18:17.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:18:17.738 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-514.el8.x86_64"... 2023-09-30T02:18:17.738 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-09-30T02:18:17.755 INFO:teuthology.orchestra.run.smithi157.stdout:4.18.0-514.el8.x86_64 2023-09-30T02:18:17.755 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T02:18:17.755 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T02:18:17.755 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T02:18:18.756 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T02:18:18.757 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T02:18:19.037 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2023-09-30T02:18:19.060 DEBUG:teuthology.parallel:result is None 2023-09-30T02:18:19.060 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T02:18:19.067 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T02:18:19.067 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T02:18:19.070 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T02:18:19.573 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T02:18:19.582 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T02:18:19.611 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T02:18:19.611 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T02:18:19.613 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T02:18:19.765 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T02:18:19.771 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T02:18:19.772 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T02:18:19.774 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T02:18:19.818 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T02:18:19.819 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T02:18:19.830 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T02:18:19.832 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T02:18:19.834 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T02:18:19.841 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T02:18:19.841 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T02:18:19.844 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T02:18:19.872 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T02:18:19.886 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T02:18:19.887 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T02:18:19.915 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T02:18:19.934 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T02:18:20.021 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T02:18:20.070 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T02:18:20.156 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T02:18:20.198 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:18:20.198 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T02:18:20.266 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T02:18:20.315 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T02:18:20.409 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T02:18:20.452 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T02:18:20.492 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:18:20.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T02:18:20.555 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2023-09-30T02:18:20.557 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2023-09-30T02:18:20.631 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:18:20.646 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:18:21.108 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T02:18:21.118 INFO:teuthology.task.internal:Starting timer... 2023-09-30T02:18:21.118 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T02:18:21.155 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T02:18:21.217 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T02:18:21.218 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2023-09-30T02:18:21.300 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2023-09-30T02:18:21.301 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2023-09-30T02:18:21.381 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2023-09-30T02:18:21.382 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T02:18:21.383 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2023-09-30T02:18:21.444 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2023-09-30T02:18:21.445 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2023-09-30T02:18:21.502 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2023-09-30T02:18:21.502 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2023-09-30T02:18:21.502 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:18:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:18:21.531 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:18:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:18:21.560 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T02:18:21.560 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2023-09-30T02:18:21.602 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2023-09-30T02:18:21.630 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T02:18:21.640 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-30T02:18:21.645 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-30T02:18:21.762 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T02:18:21.780 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T02:18:21.781 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T02:25:46.958 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2023-09-30T02:25:46.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T02:25:46.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:25:47.030 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-09-30T02:25:47.100 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-09-30T02:25:47.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:25:47.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T02:25:47.172 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-09-30T02:25:47.242 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T02:25:47.242 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T02:25:47.252 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T02:25:47.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T02:25:47.253 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:25:47.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T02:25:47.255 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:25:47.285 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T02:25:47.299 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T02:25:47.318 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T02:25:47.333 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T02:25:47.359 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2023-09-30T02:25:47.370 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2023-09-30T02:25:47.385 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T02:25:47.398 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T02:25:47.399 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2023-09-30T02:25:47.411 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2023-09-30T02:25:47.426 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T02:25:47.440 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T02:25:47.478 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-09-30T02:25:47.481 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:25:47.481 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-09-30T02:25:47.481 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.481 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.481 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.482 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.510 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-09-30T02:25:47.513 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:25:47.513 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-09-30T02:25:47.513 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.514 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.514 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.514 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:25:47.515 INFO:teuthology.run_tasks:Running task install... 2023-09-30T02:25:47.523 DEBUG:teuthology.task.install:project ceph 2023-09-30T02:25:47.524 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T02:25:47.524 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T02:25:47.524 INFO:teuthology.task.install:Using flavor: default 2023-09-30T02:25:47.529 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T02:25:47.529 INFO:teuthology.task.install:extra packages: [] 2023-09-30T02:25:47.530 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T02:25:47.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:25:47.532 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T02:25:47.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:25:47.682 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T02:25:47.682 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T02:25:47.762 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T02:25:47.762 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T02:25:47.895 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T02:25:47.895 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:25:47.895 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T02:25:47.923 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T02:25:47.924 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T02:25:47.967 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T02:25:47.967 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:25:47.967 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T02:25:47.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T02:25:47.996 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T02:25:47.996 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T02:25:48.065 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T02:25:48.069 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T02:25:48.136 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2023-09-30T02:25:48.138 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2023-09-30T02:25:48.142 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T02:25:48.208 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2023-09-30T02:25:48.210 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2023-09-30T02:25:48.407 INFO:teuthology.orchestra.run.smithi115.stdout:76 files removed 2023-09-30T02:25:48.437 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw 2023-09-30T02:25:48.477 INFO:teuthology.orchestra.run.smithi157.stdout:76 files removed 2023-09-30T02:25:48.506 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2023-09-30T02:25:49.355 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 178 kB/s | 78 kB 00:00 2023-09-30T02:25:49.424 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 134 kB/s | 78 kB 00:00 2023-09-30T02:25:49.781 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2023-09-30T02:25:49.849 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2023-09-30T02:25:50.173 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2023-09-30T02:25:50.225 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2023-09-30T02:25:51.915 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - AppStream 19 MB/s | 33 MB 00:01 2023-09-30T02:25:52.033 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 19 MB/s | 33 MB 00:01 2023-09-30T02:25:59.332 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - BaseOS 36 MB/s | 48 MB 00:01 2023-09-30T02:25:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 48 MB 00:01 2023-09-30T02:26:06.532 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2023-09-30T02:26:06.925 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 101 kB/s | 18 kB 00:00 2023-09-30T02:26:07.241 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-30T02:26:07.617 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-09-30T02:26:10.968 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - PowerTools 2.0 MB/s | 6.5 MB 00:03 2023-09-30T02:26:11.442 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 8 - PowerTools 1.6 MB/s | 6.5 MB 00:04 2023-09-30T02:26:12.419 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-09-30T02:26:12.946 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-09-30T02:26:14.217 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-09-30T02:26:14.555 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 16 MB 00:01 2023-09-30T02:26:18.111 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 422 kB/s | 24 kB 00:00 2023-09-30T02:26:18.428 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 325 kB/s | 24 kB 00:00 2023-09-30T02:26:22.584 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:26:22.586 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:26:22.586 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T02:26:22.587 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T02:26:22.588 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout:Install 21 Packages 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2023-09-30T02:26:22.589 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:26:22.590 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 63 M 2023-09-30T02:26:22.590 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:26:22.817 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:26:22.820 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T02:26:22.821 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:26:22.822 INFO:teuthology.orchestra.run.smithi115.stdout:Install 21 Packages 2023-09-30T02:26:22.823 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2023-09-30T02:26:22.823 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:26:22.823 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 63 M 2023-09-30T02:26:22.823 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:26:24.196 INFO:teuthology.orchestra.run.smithi157.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 4.6 MB/s | 5.2 MB 00:01 2023-09-30T02:26:24.296 INFO:teuthology.orchestra.run.smithi157.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 244 kB/s | 24 kB 00:00 2023-09-30T02:26:24.538 INFO:teuthology.orchestra.run.smithi157.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 3.2 MB/s | 796 kB 00:00 2023-09-30T02:26:25.121 INFO:teuthology.orchestra.run.smithi157.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 869 kB/s | 506 kB 00:00 2023-09-30T02:26:25.870 INFO:teuthology.orchestra.run.smithi115.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 2.0 MB/s | 5.2 MB 00:02 2023-09-30T02:26:25.995 INFO:teuthology.orchestra.run.smithi115.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 195 kB/s | 24 kB 00:00 2023-09-30T02:26:26.212 INFO:teuthology.orchestra.run.smithi115.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 3.6 MB/s | 796 kB 00:00 2023-09-30T02:26:26.305 INFO:teuthology.orchestra.run.smithi157.stdout:(5/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 7.2 MB/s | 23 MB 00:03 2023-09-30T02:26:26.338 INFO:teuthology.orchestra.run.smithi115.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 4.0 MB/s | 506 kB 00:00 2023-09-30T02:26:26.472 INFO:teuthology.orchestra.run.smithi157.stdout:(6/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 4.9 MB/s | 17 MB 00:03 2023-09-30T02:26:26.497 INFO:teuthology.orchestra.run.smithi157.stdout:(7/23): python3-ceph-argparse-17.2.6-1165.g93dc 233 kB/s | 45 kB 00:00 2023-09-30T02:26:26.547 INFO:teuthology.orchestra.run.smithi157.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 1.5 MB/s | 115 kB 00:00 2023-09-30T02:26:26.606 INFO:teuthology.orchestra.run.smithi157.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.7 MB/s | 190 kB 00:00 2023-09-30T02:26:26.698 INFO:teuthology.orchestra.run.smithi157.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 2.4 MB/s | 370 kB 00:00 2023-09-30T02:26:26.731 INFO:teuthology.orchestra.run.smithi157.stdout:(11/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 2.8 MB/s | 358 kB 00:00 2023-09-30T02:26:26.773 INFO:teuthology.orchestra.run.smithi157.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.4 MB/s | 107 kB 00:00 2023-09-30T02:26:26.918 INFO:teuthology.orchestra.run.smithi157.stdout:(13/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 3.1 MB/s | 5.6 MB 00:01 2023-09-30T02:26:27.204 INFO:teuthology.orchestra.run.smithi115.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 6.4 MB/s | 5.6 MB 00:00 2023-09-30T02:26:27.288 INFO:teuthology.orchestra.run.smithi115.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 537 kB/s | 45 kB 00:00 2023-09-30T02:26:27.332 INFO:teuthology.orchestra.run.smithi157.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 106 kB/s | 44 kB 00:00 2023-09-30T02:26:27.364 INFO:teuthology.orchestra.run.smithi115.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.5 MB/s | 115 kB 00:00 2023-09-30T02:26:27.438 INFO:teuthology.orchestra.run.smithi115.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.5 MB/s | 190 kB 00:00 2023-09-30T02:26:27.474 INFO:teuthology.orchestra.run.smithi157.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 332 kB/s | 47 kB 00:00 2023-09-30T02:26:27.508 INFO:teuthology.orchestra.run.smithi157.stdout:(16/23): librdkafka-0.11.4-3.el8.x86_64.rpm 456 kB/s | 354 kB 00:00 2023-09-30T02:26:27.547 INFO:teuthology.orchestra.run.smithi115.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 3.3 MB/s | 370 kB 00:00 2023-09-30T02:26:27.672 INFO:teuthology.orchestra.run.smithi115.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 2.8 MB/s | 358 kB 00:00 2023-09-30T02:26:27.716 INFO:teuthology.orchestra.run.smithi157.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2023-09-30T02:26:27.764 INFO:teuthology.orchestra.run.smithi115.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 107 kB 00:00 2023-09-30T02:26:27.808 INFO:teuthology.orchestra.run.smithi157.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 917 kB/s | 306 kB 00:00 2023-09-30T02:26:27.850 INFO:teuthology.orchestra.run.smithi157.stdout:(19/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 241 kB/s | 259 kB 00:01 2023-09-30T02:26:27.884 INFO:teuthology.orchestra.run.smithi157.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 449 kB/s | 75 kB 00:00 2023-09-30T02:26:28.084 INFO:teuthology.orchestra.run.smithi157.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 6.3 MB/s | 1.7 MB 00:00 2023-09-30T02:26:28.173 INFO:teuthology.orchestra.run.smithi115.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 867 kB/s | 354 kB 00:00 2023-09-30T02:26:28.266 INFO:teuthology.orchestra.run.smithi157.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 9.4 MB/s | 3.6 MB 00:00 2023-09-30T02:26:28.415 INFO:teuthology.orchestra.run.smithi115.stdout:(13/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 4.6 MB/s | 23 MB 00:05 2023-09-30T02:26:28.440 INFO:teuthology.orchestra.run.smithi115.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 973 kB/s | 259 kB 00:00 2023-09-30T02:26:28.468 INFO:teuthology.orchestra.run.smithi157.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 5.6 MB/s | 3.5 MB 00:00 2023-09-30T02:26:28.470 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:26:28.470 INFO:teuthology.orchestra.run.smithi157.stdout:Total 11 MB/s | 63 MB 00:05 2023-09-30T02:26:28.513 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:26:28.559 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:26:28.559 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:26:28.649 INFO:teuthology.orchestra.run.smithi115.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 188 kB/s | 44 kB 00:00 2023-09-30T02:26:28.674 INFO:teuthology.orchestra.run.smithi115.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 201 kB/s | 47 kB 00:00 2023-09-30T02:26:28.935 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:26:28.938 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:26:28.999 INFO:teuthology.orchestra.run.smithi115.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 181 kB/s | 59 kB 00:00 2023-09-30T02:26:29.149 INFO:teuthology.orchestra.run.smithi115.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 612 kB/s | 306 kB 00:00 2023-09-30T02:26:29.166 INFO:teuthology.orchestra.run.smithi115.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2023-09-30T02:26:29.408 INFO:teuthology.orchestra.run.smithi115.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2023-09-30T02:26:29.842 INFO:teuthology.orchestra.run.smithi115.stdout:(21/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 2.6 MB/s | 17 MB 00:06 2023-09-30T02:26:29.909 INFO:teuthology.orchestra.run.smithi115.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 4.7 MB/s | 3.5 MB 00:00 2023-09-30T02:26:30.059 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:26:30.067 INFO:teuthology.orchestra.run.smithi115.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 5.4 MB/s | 3.6 MB 00:00 2023-09-30T02:26:30.070 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:26:30.070 INFO:teuthology.orchestra.run.smithi115.stdout:Total 8.7 MB/s | 63 MB 00:07 2023-09-30T02:26:30.114 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:26:30.160 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:26:30.160 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:26:30.283 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T02:26:30.447 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T02:26:30.529 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:26:30.532 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:26:30.844 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T02:26:31.014 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T02:26:31.109 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T02:26:31.413 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T02:26:31.622 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T02:26:31.656 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:26:31.856 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T02:26:32.025 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T02:26:32.027 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T02:26:32.230 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T02:26:32.407 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T02:26:32.447 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T02:26:32.618 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T02:26:32.624 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T02:26:32.724 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T02:26:32.903 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T02:26:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T02:26:33.105 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T02:26:33.191 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T02:26:33.593 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T02:26:33.639 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T02:26:33.796 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T02:26:33.862 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T02:26:33.866 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T02:26:34.057 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T02:26:34.063 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T02:26:34.225 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T02:26:34.270 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T02:26:34.447 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T02:26:34.465 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T02:26:34.590 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T02:26:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T02:26:34.855 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T02:26:34.926 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T02:26:35.149 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T02:26:35.156 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T02:26:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T02:26:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T02:26:35.397 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T02:26:35.542 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T02:26:35.581 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T02:26:35.719 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T02:26:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T02:26:35.917 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T02:26:36.152 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T02:26:36.282 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T02:26:36.333 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T02:26:36.583 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T02:26:36.765 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T02:26:36.948 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T02:26:37.149 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T02:26:38.507 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T02:26:38.721 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T02:26:39.344 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T02:26:39.631 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T02:26:39.744 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T02:26:39.828 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T02:26:39.921 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T02:26:40.057 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T02:26:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T02:26:40.866 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T02:26:40.961 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T02:26:41.102 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /run 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2023-09-30T02:26:58.644 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2023-09-30T02:26:58.645 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-09-30T02:26:58.645 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2023-09-30T02:26:59.566 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2023-09-30T02:26:59.567 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2023-09-30T02:26:59.567 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2023-09-30T02:26:59.567 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-09-30T02:26:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:00.344 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T02:27:00.456 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T02:27:00.456 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T02:27:00.694 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T02:27:00.695 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T02:27:01.245 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T02:27:01.323 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T02:27:01.323 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T02:27:01.418 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T02:27:01.419 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T02:27:01.420 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T02:27:01.420 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T02:27:01.420 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T02:27:01.579 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T02:27:01.579 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T02:27:02.228 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T02:27:02.229 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T02:27:02.230 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T02:27:02.231 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T02:27:02.231 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T02:27:02.547 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T02:27:02.547 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:02.547 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2023-09-30T02:27:02.547 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.547 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.547 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:02.549 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:02.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.406 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.407 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:03.408 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:03.665 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test 2023-09-30T02:27:03.743 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:45 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:04.512 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:46 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:05.330 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:05.331 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:05.331 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:05.332 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:27:05.333 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:05.333 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2023-09-30T02:27:05.333 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:05.333 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 53 M 2023-09-30T02:27:05.333 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 233 M 2023-09-30T02:27:05.333 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:27:05.879 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 828 kB/s | 175 kB 00:00 2023-09-30T02:27:05.937 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): jq-1.6-7.el8.x86_64.rpm 751 kB/s | 202 kB 00:00 2023-09-30T02:27:06.079 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 936 kB/s | 187 kB 00:00 2023-09-30T02:27:06.099 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T02:27:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T02:27:06.102 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T02:27:06.102 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:06.102 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:27:06.102 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:06.102 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2023-09-30T02:27:06.102 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:06.103 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 53 M 2023-09-30T02:27:06.103 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 233 M 2023-09-30T02:27:06.103 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:27:06.113 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-09-30T02:27:06.139 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T02:27:06.614 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 943 kB/s | 202 kB 00:00 2023-09-30T02:27:06.673 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2023-09-30T02:27:06.757 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-09-30T02:27:06.815 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T02:27:07.032 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 276 kB/s | 175 kB 00:00 2023-09-30T02:27:08.222 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 52 MB 00:02 2023-09-30T02:27:08.224 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:08.224 INFO:teuthology.orchestra.run.smithi157.stdout:Total 18 MB/s | 53 MB 00:02 2023-09-30T02:27:08.269 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:27:08.290 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:27:08.291 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:27:08.624 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:27:08.627 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:27:09.100 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:27:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T02:27:09.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T02:27:09.716 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T02:27:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T02:27:10.023 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T02:27:10.190 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T02:27:11.890 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 9.4 MB/s | 52 MB 00:05 2023-09-30T02:27:11.891 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:11.892 INFO:teuthology.orchestra.run.smithi115.stdout:Total 9.1 MB/s | 53 MB 00:05 2023-09-30T02:27:11.936 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:27:11.957 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:27:11.957 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:27:12.274 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:27:12.277 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:27:12.744 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:27:12.951 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T02:27:13.168 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T02:27:13.282 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T02:27:13.360 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T02:27:13.539 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T02:27:13.723 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T02:27:15.132 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T02:27:15.371 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T02:27:15.660 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T02:27:15.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T02:27:15.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T02:27:15.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T02:27:15.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T02:27:15.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T02:27:16.302 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T02:27:16.302 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:16.302 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:27:16.302 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:16.302 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-7.el8.x86_64 2023-09-30T02:27:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T02:27:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T02:27:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T02:27:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:16.303 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:16.537 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2023-09-30T02:27:17.399 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:18.714 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T02:27:18.937 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T02:27:18.973 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:18.977 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:27:18.977 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:18.977 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:27:18.977 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T02:27:18.977 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T02:27:18.978 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T02:27:18.979 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T02:27:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T02:27:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T02:27:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T02:27:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T02:27:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T02:27:18.980 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T02:27:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T02:27:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T02:27:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T02:27:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T02:27:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T02:27:18.981 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:18.982 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:27:18.983 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:18.983 INFO:teuthology.orchestra.run.smithi157.stdout:Install 32 Packages 2023-09-30T02:27:18.983 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:18.984 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 28 M 2023-09-30T02:27:18.984 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 110 M 2023-09-30T02:27:18.984 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:27:19.223 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T02:27:19.223 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T02:27:19.223 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T02:27:19.223 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T02:27:19.223 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T02:27:19.223 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T02:27:19.647 INFO:teuthology.orchestra.run.smithi157.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 41 kB/s | 6.4 kB 00:00 2023-09-30T02:27:19.909 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T02:27:19.909 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.6-7.el8.x86_64 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:19.910 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:20.158 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph 2023-09-30T02:27:20.464 INFO:teuthology.orchestra.run.smithi157.stdout:(2/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 2.4 MB/s | 2.3 MB 00:00 2023-09-30T02:27:20.623 INFO:teuthology.orchestra.run.smithi157.stdout:(3/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.4 MB/s | 1.6 MB 00:01 2023-09-30T02:27:20.789 INFO:teuthology.orchestra.run.smithi157.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 1.4 MB/s | 238 kB 00:00 2023-09-30T02:27:20.889 INFO:teuthology.orchestra.run.smithi157.stdout:(5/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 3.2 MB/s | 3.9 MB 00:01 2023-09-30T02:27:21.011 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:03 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:21.023 INFO:teuthology.orchestra.run.smithi157.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 287 kB/s | 39 kB 00:00 2023-09-30T02:27:21.049 INFO:teuthology.orchestra.run.smithi157.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 606 kB/s | 157 kB 00:00 2023-09-30T02:27:21.107 INFO:teuthology.orchestra.run.smithi157.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 977 kB/s | 57 kB 00:00 2023-09-30T02:27:21.124 INFO:teuthology.orchestra.run.smithi157.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2023-09-30T02:27:21.225 INFO:teuthology.orchestra.run.smithi157.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-09-30T02:27:21.325 INFO:teuthology.orchestra.run.smithi157.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2023-09-30T02:27:21.384 INFO:teuthology.orchestra.run.smithi157.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2023-09-30T02:27:21.450 INFO:teuthology.orchestra.run.smithi157.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 2.8 MB/s | 641 kB 00:00 2023-09-30T02:27:21.534 INFO:teuthology.orchestra.run.smithi157.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 290 kB/s | 44 kB 00:00 2023-09-30T02:27:21.601 INFO:teuthology.orchestra.run.smithi157.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.5 MB/s | 173 kB 00:00 2023-09-30T02:27:21.627 INFO:teuthology.orchestra.run.smithi157.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2023-09-30T02:27:21.669 INFO:teuthology.orchestra.run.smithi157.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 5.6 MB/s | 384 kB 00:00 2023-09-30T02:27:21.686 INFO:teuthology.orchestra.run.smithi157.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2023-09-30T02:27:21.711 INFO:teuthology.orchestra.run.smithi157.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 420 kB/s | 18 kB 00:00 2023-09-30T02:27:21.736 INFO:teuthology.orchestra.run.smithi157.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 967 kB/s | 49 kB 00:00 2023-09-30T02:27:21.761 INFO:teuthology.orchestra.run.smithi157.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-09-30T02:27:21.824 INFO:teuthology.orchestra.run.smithi157.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.2 MB/s | 283 kB 00:00 2023-09-30T02:27:21.853 INFO:teuthology.orchestra.run.smithi157.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2023-09-30T02:27:21.881 INFO:teuthology.orchestra.run.smithi157.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 339 kB/s | 19 kB 00:00 2023-09-30T02:27:21.896 INFO:teuthology.orchestra.run.smithi157.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 570 kB/s | 24 kB 00:00 2023-09-30T02:27:21.921 INFO:teuthology.orchestra.run.smithi157.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 698 kB/s | 29 kB 00:00 2023-09-30T02:27:21.938 INFO:teuthology.orchestra.run.smithi157.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 657 kB/s | 27 kB 00:00 2023-09-30T02:27:21.972 INFO:teuthology.orchestra.run.smithi157.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 5.0 MB/s | 255 kB 00:00 2023-09-30T02:27:21.997 INFO:teuthology.orchestra.run.smithi157.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2023-09-30T02:27:22.022 INFO:teuthology.orchestra.run.smithi157.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-09-30T02:27:22.047 INFO:teuthology.orchestra.run.smithi157.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 456 kB/s | 23 kB 00:00 2023-09-30T02:27:22.573 INFO:teuthology.orchestra.run.smithi157.stdout:(32/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 7.9 MB/s | 17 MB 00:02 2023-09-30T02:27:22.576 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:22.577 INFO:teuthology.orchestra.run.smithi157.stdout:Total 7.9 MB/s | 28 MB 00:03 2023-09-30T02:27:22.604 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T02:27:22.608 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T02:27:22.609 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T02:27:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout:Install 32 Packages 2023-09-30T02:27:22.611 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:22.612 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 28 M 2023-09-30T02:27:22.612 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 110 M 2023-09-30T02:27:22.612 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:27:22.662 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:27:22.698 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:27:22.698 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:27:22.985 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:27:22.988 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:27:23.532 INFO:teuthology.orchestra.run.smithi115.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 29 kB/s | 6.4 kB 00:00 2023-09-30T02:27:24.049 INFO:teuthology.orchestra.run.smithi115.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 1.6 MB 00:00 2023-09-30T02:27:24.143 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:27:24.216 INFO:teuthology.orchestra.run.smithi115.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 2.5 MB/s | 2.3 MB 00:00 2023-09-30T02:27:24.343 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T02:27:24.349 INFO:teuthology.orchestra.run.smithi115.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 1.7 MB/s | 238 kB 00:00 2023-09-30T02:27:24.550 INFO:teuthology.orchestra.run.smithi115.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 785 kB/s | 157 kB 00:00 2023-09-30T02:27:24.563 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T02:27:24.634 INFO:teuthology.orchestra.run.smithi115.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 465 kB/s | 39 kB 00:00 2023-09-30T02:27:24.692 INFO:teuthology.orchestra.run.smithi115.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2023-09-30T02:27:24.751 INFO:teuthology.orchestra.run.smithi115.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 978 kB/s | 57 kB 00:00 2023-09-30T02:27:24.805 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T02:27:24.851 INFO:teuthology.orchestra.run.smithi115.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.5 MB/s | 457 kB 00:00 2023-09-30T02:27:25.036 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T02:27:25.052 INFO:teuthology.orchestra.run.smithi115.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2023-09-30T02:27:25.181 INFO:teuthology.orchestra.run.smithi115.stdout:(11/32): python3-cryptography-3.2.1-6.el8.x86_6 4.9 MB/s | 641 kB 00:00 2023-09-30T02:27:25.201 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T02:27:25.236 INFO:teuthology.orchestra.run.smithi115.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.9 MB/s | 109 kB 00:00 2023-09-30T02:27:25.406 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T02:27:25.574 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T02:27:25.703 INFO:teuthology.orchestra.run.smithi115.stdout:(13/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 1.8 MB/s | 3.9 MB 00:02 2023-09-30T02:27:25.778 INFO:teuthology.orchestra.run.smithi115.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 80 kB/s | 44 kB 00:00 2023-09-30T02:27:25.802 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T02:27:26.036 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T02:27:26.045 INFO:teuthology.orchestra.run.smithi115.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 647 kB/s | 173 kB 00:00 2023-09-30T02:27:26.239 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T02:27:26.454 INFO:teuthology.orchestra.run.smithi115.stdout:(16/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 7.0 MB/s | 17 MB 00:02 2023-09-30T02:27:26.479 INFO:teuthology.orchestra.run.smithi115.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 239 kB/s | 185 kB 00:00 2023-09-30T02:27:26.494 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T02:27:26.554 INFO:teuthology.orchestra.run.smithi115.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 756 kB/s | 384 kB 00:00 2023-09-30T02:27:26.579 INFO:teuthology.orchestra.run.smithi115.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-09-30T02:27:26.663 INFO:teuthology.orchestra.run.smithi115.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 469 kB/s | 49 kB 00:00 2023-09-30T02:27:26.688 INFO:teuthology.orchestra.run.smithi115.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2023-09-30T02:27:26.759 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T02:27:26.797 INFO:teuthology.orchestra.run.smithi115.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-09-30T02:27:26.847 INFO:teuthology.orchestra.run.smithi115.stdout:(23/32): python3-jaraco-6.2-6.el8.noarch.rpm 28 kB/s | 11 kB 00:00 2023-09-30T02:27:26.897 INFO:teuthology.orchestra.run.smithi115.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-09-30T02:27:26.948 INFO:teuthology.orchestra.run.smithi115.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-09-30T02:27:27.006 INFO:teuthology.orchestra.run.smithi115.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2023-09-30T02:27:27.048 INFO:teuthology.orchestra.run.smithi115.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2023-09-30T02:27:27.091 INFO:teuthology.orchestra.run.smithi115.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 663 kB/s | 283 kB 00:00 2023-09-30T02:27:27.168 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T02:27:27.282 INFO:teuthology.orchestra.run.smithi115.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 446 kB/s | 85 kB 00:00 2023-09-30T02:27:27.349 INFO:teuthology.orchestra.run.smithi115.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 746 kB/s | 255 kB 00:00 2023-09-30T02:27:27.374 INFO:teuthology.orchestra.run.smithi115.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2023-09-30T02:27:27.399 INFO:teuthology.orchestra.run.smithi115.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 716 kB/s | 251 kB 00:00 2023-09-30T02:27:27.404 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:27.404 INFO:teuthology.orchestra.run.smithi115.stdout:Total 5.9 MB/s | 28 MB 00:04 2023-09-30T02:27:27.441 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T02:27:27.489 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:27:27.525 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:27:27.526 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:27:27.642 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T02:27:27.841 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:27:27.844 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:27:27.864 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T02:27:28.089 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T02:27:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T02:27:28.676 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T02:27:28.898 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T02:27:28.967 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:27:29.093 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T02:27:29.143 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T02:27:29.353 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T02:27:29.378 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T02:27:29.553 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T02:27:29.641 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T02:27:29.751 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T02:27:29.919 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T02:27:29.958 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T02:27:30.163 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T02:27:30.265 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T02:27:30.331 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T02:27:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T02:27:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T02:27:30.758 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T02:27:30.979 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T02:27:31.019 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T02:27:31.220 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T02:27:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T02:27:31.424 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T02:27:31.812 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T02:27:32.048 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T02:27:32.261 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T02:27:32.476 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T02:27:32.670 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T02:27:32.754 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T02:27:32.956 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T02:27:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T02:27:33.184 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T02:27:33.349 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T02:27:33.414 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T02:27:33.545 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T02:27:33.599 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T02:27:33.762 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T02:27:33.835 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T02:27:34.089 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T02:27:34.330 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T02:27:34.421 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T02:27:34.442 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T02:27:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T02:27:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T02:27:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T02:27:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T02:27:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T02:27:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T02:27:35.659 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T02:27:36.009 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T02:27:36.082 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T02:27:36.082 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:36.082 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:27:36.082 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:36.082 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:36.082 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T02:27:36.083 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T02:27:36.084 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:36.085 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:36.195 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T02:27:36.318 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2023-09-30T02:27:37.173 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:19 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:37.680 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T02:27:37.870 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T02:27:38.263 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T02:27:38.435 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T02:27:38.568 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:27:38.652 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T02:27:38.754 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:38.755 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:27:38.756 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:38.818 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T02:27:38.837 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T02:27:38.845 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2023-09-30T02:27:38.951 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T02:27:39.343 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T02:27:39.344 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T02:27:39.345 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T02:27:39.694 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:21 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:40.492 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T02:27:40.492 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T02:27:40.493 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T02:27:40.494 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:40.495 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:40.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base 2023-09-30T02:27:41.282 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:27:41.283 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:41.284 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 80 k 2023-09-30T02:27:41.284 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 363 k 2023-09-30T02:27:41.284 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:27:41.543 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:23 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:41.794 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 157 kB/s | 80 kB 00:00 2023-09-30T02:27:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:Total 157 kB/s | 80 kB 00:00 2023-09-30T02:27:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:27:41.803 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:27:41.803 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:27:41.805 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:27:41.808 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:27:41.849 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:27:42.397 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:42.542 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:42.775 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:42.946 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:27:43.133 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:43.134 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:27:43.134 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:43.271 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install cephadm 2023-09-30T02:27:43.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:43.408 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:43.408 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:27:43.408 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:43.408 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:43.408 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:43.609 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T02:27:44.116 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:44.474 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:45.700 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:45.700 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:45.700 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:27:45.700 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:45.700 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:27:45.701 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T02:27:45.701 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:45.701 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:27:45.701 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:45.701 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:27:45.701 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:45.702 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 80 k 2023-09-30T02:27:45.702 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 363 k 2023-09-30T02:27:45.702 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:27:46.080 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:46.081 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:46.081 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T02:27:46.081 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:27:46.082 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:46.083 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 161 k 2023-09-30T02:27:46.083 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 490 k 2023-09-30T02:27:46.083 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:27:46.519 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 98 kB/s | 80 kB 00:00 2023-09-30T02:27:46.519 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:46.520 INFO:teuthology.orchestra.run.smithi115.stdout:Total 98 kB/s | 80 kB 00:00 2023-09-30T02:27:46.520 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:27:46.529 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:27:46.529 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:27:46.531 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:27:46.535 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:27:46.715 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 255 kB/s | 161 kB 00:00 2023-09-30T02:27:46.716 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:46.717 INFO:teuthology.orchestra.run.smithi157.stdout:Total 254 kB/s | 161 kB 00:00 2023-09-30T02:27:46.717 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:27:46.729 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:27:46.729 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:27:46.780 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:27:46.783 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:27:46.850 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:27:46.890 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:27:47.127 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T02:27:47.311 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:47.489 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:47.563 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T02:27:47.723 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:48.189 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T02:27:48.189 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:48.189 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:27:48.190 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:48.190 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:48.190 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:48.322 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:27:48.322 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:48.323 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:27:48.323 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:48.323 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:48.323 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:48.429 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2023-09-30T02:27:48.530 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T02:27:49.291 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:49.386 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:50.700 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:27:50.893 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:50.894 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:27:50.895 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:50.988 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:50.988 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:50.988 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T02:27:50.988 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:50.988 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:27:50.989 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T02:27:50.989 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:50.989 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:27:50.989 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:50.989 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:27:50.989 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:50.990 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 161 k 2023-09-30T02:27:50.990 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 490 k 2023-09-30T02:27:50.990 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:27:50.991 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T02:27:51.619 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 257 kB/s | 161 kB 00:00 2023-09-30T02:27:51.619 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:51.620 INFO:teuthology.orchestra.run.smithi115.stdout:Total 256 kB/s | 161 kB 00:00 2023-09-30T02:27:51.620 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:27:51.631 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:27:51.631 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:27:51.681 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:27:51.684 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:27:51.805 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:27:51.852 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:33 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:52.738 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T02:27:53.159 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T02:27:53.428 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:27:53.429 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:53.429 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:53.430 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:27:53.431 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:27:53.431 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2023-09-30T02:27:53.431 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:53.431 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.0 M 2023-09-30T02:27:53.431 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 8.0 M 2023-09-30T02:27:53.431 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:27:53.759 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T02:27:53.759 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:53.759 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:27:53.759 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:27:53.759 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:53.759 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:53.974 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr 2023-09-30T02:27:54.018 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 58 kB/s | 23 kB 00:00 2023-09-30T02:27:54.043 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 34 kB/s | 14 kB 00:00 2023-09-30T02:27:54.119 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 448 kB/s | 34 kB 00:00 2023-09-30T02:27:54.169 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 283 kB/s | 43 kB 00:00 2023-09-30T02:27:54.219 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.9 MB/s | 196 kB 00:00 2023-09-30T02:27:54.826 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:55.078 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 1.2 MB/s | 1.7 MB 00:01 2023-09-30T02:27:55.080 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:27:55.080 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2023-09-30T02:27:55.092 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:27:55.102 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:27:55.102 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:27:55.184 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:27:55.187 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:27:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:27:55.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T02:27:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T02:27:56.144 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T02:27:56.215 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:27:56.281 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T02:27:56.409 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:56.410 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:27:56.410 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:27:56.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T02:27:56.591 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T02:27:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T02:27:57.237 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T02:27:57.237 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T02:27:57.238 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T02:27:57.238 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T02:27:57.238 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T02:27:57.238 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T02:27:57.367 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T02:27:57.869 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:27:57.870 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:27:57.992 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T02:27:58.845 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:27:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:27:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:58.951 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T02:27:58.952 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T02:27:58.953 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:27:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:27:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2023-09-30T02:27:58.953 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:27:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 2.0 M 2023-09-30T02:27:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 8.0 M 2023-09-30T02:27:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:27:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 55 kB/s | 23 kB 00:00 2023-09-30T02:27:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 32 kB/s | 14 kB 00:00 2023-09-30T02:27:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 284 kB/s | 43 kB 00:00 2023-09-30T02:27:59.964 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2023-09-30T02:28:00.164 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 841 kB/s | 196 kB 00:00 2023-09-30T02:28:00.431 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:00.432 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-09-30T02:28:00.432 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:28:00.432 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-09-30T02:28:00.432 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:00.432 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T02:28:00.433 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:00.434 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:00.434 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================================ 2023-09-30T02:28:00.434 INFO:teuthology.orchestra.run.smithi157.stdout:Install 8 Packages 2023-09-30T02:28:00.434 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:00.435 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 35 M 2023-09-30T02:28:00.435 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 210 M 2023-09-30T02:28:00.435 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 1.2 MB/s | 1.7 MB 00:01 2023-09-30T02:28:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2023-09-30T02:28:00.779 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:00.789 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:00.789 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:00.888 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:00.891 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:01.092 INFO:teuthology.orchestra.run.smithi157.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2023-09-30T02:28:01.155 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:01.184 INFO:teuthology.orchestra.run.smithi157.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.4 MB/s | 4.9 MB 00:00 2023-09-30T02:28:01.259 INFO:teuthology.orchestra.run.smithi157.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2023-09-30T02:28:01.359 INFO:teuthology.orchestra.run.smithi157.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2023-09-30T02:28:01.369 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T02:28:01.564 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T02:28:01.710 INFO:teuthology.orchestra.run.smithi157.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 30 MB/s | 14 MB 00:00 2023-09-30T02:28:01.735 INFO:teuthology.orchestra.run.smithi157.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.7 MB/s | 645 kB 00:00 2023-09-30T02:28:01.740 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T02:28:01.768 INFO:teuthology.orchestra.run.smithi157.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2023-09-30T02:28:01.894 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T02:28:02.195 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T02:28:02.389 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T02:28:02.678 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T02:28:02.679 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T02:28:02.679 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T02:28:02.679 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T02:28:02.679 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T02:28:02.679 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T02:28:02.979 INFO:teuthology.orchestra.run.smithi157.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 3.2 MB/s | 7.4 MB 00:02 2023-09-30T02:28:02.980 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:02.980 INFO:teuthology.orchestra.run.smithi157.stdout:Total 14 MB/s | 35 MB 00:02 2023-09-30T02:28:03.216 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:03.235 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:03.235 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:03.332 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T02:28:03.332 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:03.332 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:03.333 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:03.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T02:28:03.611 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:03.614 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:04.170 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:04.316 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:04.373 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T02:28:04.503 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T02:28:04.681 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T02:28:05.280 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T02:28:05.515 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T02:28:05.905 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================================ 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================================ 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:28:05.906 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================================ 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout:Install 8 Packages 2023-09-30T02:28:05.907 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:05.908 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 35 M 2023-09-30T02:28:05.908 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 210 M 2023-09-30T02:28:05.908 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:06.141 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T02:28:06.315 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T02:28:06.722 INFO:teuthology.orchestra.run.smithi115.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2023-09-30T02:28:06.788 INFO:teuthology.orchestra.run.smithi115.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.8 MB/s | 4.9 MB 00:00 2023-09-30T02:28:06.864 INFO:teuthology.orchestra.run.smithi115.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2023-09-30T02:28:06.955 INFO:teuthology.orchestra.run.smithi115.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2023-09-30T02:28:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T02:28:07.306 INFO:teuthology.orchestra.run.smithi115.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 31 MB/s | 14 MB 00:00 2023-09-30T02:28:07.331 INFO:teuthology.orchestra.run.smithi115.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.7 MB/s | 645 kB 00:00 2023-09-30T02:28:07.352 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T02:28:07.490 INFO:teuthology.orchestra.run.smithi115.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 940 kB/s | 172 kB 00:00 2023-09-30T02:28:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 5.4 MB/s | 7.4 MB 00:01 2023-09-30T02:28:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:07.657 INFO:teuthology.orchestra.run.smithi115.stdout:Total 20 MB/s | 35 MB 00:01 2023-09-30T02:28:07.889 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:07.907 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:07.907 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:07.931 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T02:28:08.334 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:08.337 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:09.734 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T02:28:09.806 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:10.003 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T02:28:10.122 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T02:28:10.277 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T02:28:10.875 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T02:28:10.972 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T02:28:11.069 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T02:28:11.230 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T02:28:11.552 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T02:28:11.553 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T02:28:11.717 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T02:28:11.869 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T02:28:12.190 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T02:28:12.191 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:12.192 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:12.307 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2023-09-30T02:28:12.571 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T02:28:12.889 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T02:28:13.127 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T02:28:13.165 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:55 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:14.762 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T02:28:14.764 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:14.765 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:14.766 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2023-09-30T02:28:14.766 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:14.766 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 1.9 M 2023-09-30T02:28:14.766 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 25 M 2023-09-30T02:28:14.766 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:14.906 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T02:28:15.344 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 129 kB/s | 20 kB 00:00 2023-09-30T02:28:15.361 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 155 kB/s | 27 kB 00:00 2023-09-30T02:28:15.487 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 893 kB/s | 126 kB 00:00 2023-09-30T02:28:15.512 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 731 kB/s | 110 kB 00:00 2023-09-30T02:28:15.637 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 106 kB/s | 47 kB 00:00 2023-09-30T02:28:15.654 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 300 kB/s | 43 kB 00:00 2023-09-30T02:28:15.696 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 741 kB/s | 155 kB 00:00 2023-09-30T02:28:16.013 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 45 kB/s | 16 kB 00:00 2023-09-30T02:28:16.030 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 84 kB/s | 33 kB 00:00 2023-09-30T02:28:16.146 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T02:28:16.163 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 174 kB/s | 81 kB 00:00 2023-09-30T02:28:16.256 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 285 kB/s | 64 kB 00:00 2023-09-30T02:28:16.306 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 432 kB/s | 61 kB 00:00 2023-09-30T02:28:16.376 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T02:28:16.572 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2023-09-30T02:28:16.575 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:16.575 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-09-30T02:28:16.636 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:16.652 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:16.653 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:16.683 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T02:28:16.684 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T02:28:16.740 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:16.743 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:17.253 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:17.330 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T02:28:17.330 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:17.331 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:17.452 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-rook 2023-09-30T02:28:17.490 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T02:28:17.679 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T02:28:17.931 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T02:28:18.263 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T02:28:18.304 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:00 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:18.743 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T02:28:19.000 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T02:28:19.309 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T02:28:19.567 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T02:28:19.880 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:19.882 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:28:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:19.882 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T02:28:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T02:28:19.883 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:19.884 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2023-09-30T02:28:19.885 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:19.885 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 1.9 M 2023-09-30T02:28:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 25 M 2023-09-30T02:28:19.886 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:20.314 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 146 kB/s | 20 kB 00:00 2023-09-30T02:28:20.355 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 149 kB/s | 27 kB 00:00 2023-09-30T02:28:20.381 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 232 kB/s | 47 kB 00:00 2023-09-30T02:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-09-30T02:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 875 kB/s | 110 kB 00:00 2023-09-30T02:28:20.548 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 926 kB/s | 155 kB 00:00 2023-09-30T02:28:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T02:28:20.582 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 300 kB/s | 43 kB 00:00 2023-09-30T02:28:20.624 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2023-09-30T02:28:20.699 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 109 kB/s | 16 kB 00:00 2023-09-30T02:28:20.750 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 499 kB/s | 81 kB 00:00 2023-09-30T02:28:20.775 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 854 kB/s | 64 kB 00:00 2023-09-30T02:28:20.800 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-09-30T02:28:20.834 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.7 MB/s | 1.2 MB 00:00 2023-09-30T02:28:20.836 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:20.836 INFO:teuthology.orchestra.run.smithi115.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2023-09-30T02:28:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T02:28:20.891 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:20.906 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:20.907 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:20.991 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:21.137 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T02:28:21.355 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T02:28:21.482 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T02:28:21.736 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T02:28:21.847 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T02:28:21.847 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T02:28:21.847 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T02:28:21.847 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T02:28:21.847 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T02:28:21.847 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T02:28:21.848 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T02:28:21.925 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T02:28:22.161 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T02:28:22.390 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T02:28:22.526 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T02:28:22.527 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:22.528 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:22.589 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T02:28:22.641 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T02:28:22.845 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T02:28:23.120 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T02:28:23.370 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T02:28:23.499 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:05 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:24.380 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T02:28:24.639 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T02:28:24.856 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T02:28:25.059 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T02:28:25.085 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:25.086 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:25.086 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:28:25.086 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:25.086 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:25.086 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T02:28:25.086 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout:Install 5 Packages 2023-09-30T02:28:25.087 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:25.088 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.0 M 2023-09-30T02:28:25.088 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2023-09-30T02:28:25.088 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:25.233 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T02:28:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T02:28:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T02:28:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T02:28:25.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T02:28:25.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T02:28:25.804 INFO:teuthology.orchestra.run.smithi157.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 373 kB/s | 121 kB 00:00 2023-09-30T02:28:25.896 INFO:teuthology.orchestra.run.smithi157.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.3 MB/s | 538 kB 00:00 2023-09-30T02:28:26.088 INFO:teuthology.orchestra.run.smithi157.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 314 kB/s | 60 kB 00:00 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T02:28:26.221 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:26.222 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:26.313 INFO:teuthology.orchestra.run.smithi157.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 935 kB/s | 476 kB 00:00 2023-09-30T02:28:26.348 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T02:28:26.787 INFO:teuthology.orchestra.run.smithi157.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 3.7 MB/s | 4.8 MB 00:01 2023-09-30T02:28:26.787 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:26.787 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.5 MB/s | 6.0 MB 00:01 2023-09-30T02:28:26.843 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:26.854 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:26.855 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:26.961 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:26.964 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:27.205 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:09 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:27.300 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:27.642 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T02:28:28.617 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T02:28:28.787 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:28.788 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:28.788 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:28:28.788 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:28.788 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:28.788 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout:Install 5 Packages 2023-09-30T02:28:28.789 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:28.790 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 6.0 M 2023-09-30T02:28:28.790 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 26 M 2023-09-30T02:28:28.790 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:28.999 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T02:28:29.296 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T02:28:29.325 INFO:teuthology.orchestra.run.smithi115.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 650 kB/s | 121 kB 00:00 2023-09-30T02:28:29.450 INFO:teuthology.orchestra.run.smithi115.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2023-09-30T02:28:29.527 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T02:28:29.533 INFO:teuthology.orchestra.run.smithi115.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.2 MB/s | 476 kB 00:00 2023-09-30T02:28:29.584 INFO:teuthology.orchestra.run.smithi115.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 451 kB/s | 60 kB 00:00 2023-09-30T02:28:29.808 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T02:28:29.808 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T02:28:29.808 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T02:28:29.808 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T02:28:29.808 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T02:28:30.209 INFO:teuthology.orchestra.run.smithi115.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 4.5 MB/s | 4.8 MB 00:01 2023-09-30T02:28:30.209 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:30.209 INFO:teuthology.orchestra.run.smithi115.stdout:Total 4.2 MB/s | 6.0 MB 00:01 2023-09-30T02:28:30.264 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:30.275 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:30.275 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:30.379 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:30.425 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T02:28:30.425 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:30.425 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:30.426 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:30.547 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2023-09-30T02:28:30.719 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:31.012 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T02:28:31.409 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:13 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:31.980 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T02:28:32.327 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T02:28:32.600 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T02:28:32.806 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T02:28:32.996 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:32.997 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:32.997 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:28:32.997 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:32.997 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:32.997 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T02:28:32.997 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 957 k 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 3.0 M 2023-09-30T02:28:32.998 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:33.081 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T02:28:33.082 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T02:28:33.082 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T02:28:33.082 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T02:28:33.082 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T02:28:33.362 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 460 kB/s | 83 kB 00:00 2023-09-30T02:28:33.454 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 3.1 MB/s | 874 kB 00:00 2023-09-30T02:28:33.455 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:33.455 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.0 MB/s | 957 kB 00:00 2023-09-30T02:28:33.466 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:33.487 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:33.487 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:33.575 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:33.578 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:33.680 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:33.744 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:33.811 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse 2023-09-30T02:28:34.056 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T02:28:34.318 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T02:28:34.670 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:16 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:34.721 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T02:28:34.722 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T02:28:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T02:28:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:35.341 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T02:28:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:35.342 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:35.568 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2023-09-30T02:28:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2023-09-30T02:28:36.256 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 957 k 2023-09-30T02:28:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 3.0 M 2023-09-30T02:28:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:36.442 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:18 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:36.625 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 464 kB/s | 83 kB 00:00 2023-09-30T02:28:36.916 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 1.8 MB/s | 874 kB 00:00 2023-09-30T02:28:36.917 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:36.917 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.4 MB/s | 957 kB 00:00 2023-09-30T02:28:36.928 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:36.948 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:36.948 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:37.033 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:37.036 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:37.215 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:37.479 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T02:28:37.688 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T02:28:37.969 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:37.970 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:37.971 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:28:37.971 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:37.971 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 267 k 2023-09-30T02:28:37.971 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.3 M 2023-09-30T02:28:37.971 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:38.088 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T02:28:38.088 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T02:28:38.333 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 738 kB/s | 267 kB 00:00 2023-09-30T02:28:38.333 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:38.334 INFO:teuthology.orchestra.run.smithi157.stdout:Total 735 kB/s | 267 kB 00:00 2023-09-30T02:28:38.334 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:38.344 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:38.344 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:38.375 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:38.378 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:38.564 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T02:28:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:38.734 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T02:28:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:38.734 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:38.837 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:28:38.991 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-volume 2023-09-30T02:28:39.280 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:28:39.854 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:21 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:40.709 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:28:40.710 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:40.710 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:40.710 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:40.710 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:40.710 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:41.421 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:28:41.422 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:41.423 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 267 k 2023-09-30T02:28:41.423 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 1.3 M 2023-09-30T02:28:41.423 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:41.628 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.3 MB/s | 267 kB 00:00 2023-09-30T02:28:41.629 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:41.629 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.3 MB/s | 267 kB 00:00 2023-09-30T02:28:41.629 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:41.639 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:41.639 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:41.686 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:41.689 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:41.871 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:41.936 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2023-09-30T02:28:42.104 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:28:42.515 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:28:42.818 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:24 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:43.156 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T02:28:43.156 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:43.156 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:43.156 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T02:28:43.156 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:43.156 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:43.344 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel 2023-09-30T02:28:44.209 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:26 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:44.411 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:44.411 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:44.411 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:28:44.412 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:44.413 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 130 k 2023-09-30T02:28:44.413 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 464 k 2023-09-30T02:28:44.413 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:44.715 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 430 kB/s | 130 kB 00:00 2023-09-30T02:28:44.716 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:44.716 INFO:teuthology.orchestra.run.smithi157.stdout:Total 427 kB/s | 130 kB 00:00 2023-09-30T02:28:44.717 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:44.722 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:44.722 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:44.772 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:44.775 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:44.882 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:45.441 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:45.782 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:45.782 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:45.782 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:28:45.783 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:45.784 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 130 k 2023-09-30T02:28:45.784 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 464 k 2023-09-30T02:28:45.784 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:46.074 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:46.075 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:46.075 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:46.075 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:28:46.075 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:46.075 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:46.288 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2023-09-30T02:28:46.569 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 165 kB/s | 130 kB 00:00 2023-09-30T02:28:46.570 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:46.570 INFO:teuthology.orchestra.run.smithi115.stdout:Total 165 kB/s | 130 kB 00:00 2023-09-30T02:28:46.570 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:46.577 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:46.577 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:46.632 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:46.637 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:47.156 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:29 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:47.402 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:47.703 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:48.339 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:48.339 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:48.339 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:48.340 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:28:48.340 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:48.340 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:48.535 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:28:48.564 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2 2023-09-30T02:28:48.720 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:48.722 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:28:48.722 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:48.825 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2023-09-30T02:28:49.425 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:31 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:49.687 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:31 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:50.818 INFO:teuthology.orchestra.run.smithi115.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:28:51.004 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:51.006 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:28:51.006 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:51.103 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs-devel 2023-09-30T02:28:51.277 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:51.278 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:51.278 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T02:28:51.278 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:51.278 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:28:51.279 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T02:28:51.279 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:51.279 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:28:51.279 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:28:51.279 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:28:51.280 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:51.280 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 25 k 2023-09-30T02:28:51.280 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 109 k 2023-09-30T02:28:51.280 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:28:51.427 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 167 kB/s | 25 kB 00:00 2023-09-30T02:28:51.427 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:51.427 INFO:teuthology.orchestra.run.smithi157.stdout:Total 166 kB/s | 25 kB 00:00 2023-09-30T02:28:51.428 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:28:51.431 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:28:51.431 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:28:51.433 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:28:51.436 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:28:51.491 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:28:51.739 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:51.960 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:33 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:52.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:52.955 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:52.955 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:28:52.956 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:28:52.956 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:28:52.956 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:53.065 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2023-09-30T02:28:53.539 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:53.540 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:53.540 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T02:28:53.540 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:53.540 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 25 k 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 109 k 2023-09-30T02:28:53.541 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:28:53.697 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 159 kB/s | 25 kB 00:00 2023-09-30T02:28:53.697 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:28:53.698 INFO:teuthology.orchestra.run.smithi115.stdout:Total 158 kB/s | 25 kB 00:00 2023-09-30T02:28:53.698 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:28:53.701 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:28:53.701 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:28:53.703 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:28:53.706 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:28:53.754 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:28:53.923 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:53.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:54.801 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:28:54.801 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:54.802 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:28:54.802 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:28:54.802 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:28:54.802 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:54.907 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2 2023-09-30T02:28:55.313 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:28:55.500 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:55.501 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:28:55.501 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:55.600 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2023-09-30T02:28:55.758 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:37 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:56.464 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:38 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:57.162 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:28:57.348 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:57.349 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:28:57.349 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:57.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1 2023-09-30T02:28:57.849 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:28:58.037 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:28:58.038 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:28:58.038 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:28:58.134 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2023-09-30T02:28:58.304 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:40 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:58.990 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:40 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:28:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:28:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:28:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:28:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:28:59.995 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados 2023-09-30T02:29:00.387 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:00.576 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:00.578 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:29:00.578 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:00.678 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2023-09-30T02:29:00.850 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:42 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:01.534 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:43 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:02.249 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:02.436 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:02.437 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:29:02.437 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:02.534 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw 2023-09-30T02:29:02.945 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:03.132 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:03.133 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:29:03.133 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:03.250 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2023-09-30T02:29:03.391 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:45 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:04.121 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:46 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:04.758 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:04.947 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:04.948 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:29:04.949 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:05.050 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs 2023-09-30T02:29:05.618 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:05.806 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:05.808 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:29:05.808 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:05.907 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:05.914 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2023-09-30T02:29:06.782 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:07.297 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:07.484 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:07.485 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:29:07.485 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:07.583 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd 2023-09-30T02:29:08.174 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:08.361 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:08.362 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2023-09-30T02:29:08.363 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:08.439 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:50 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:08.511 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2023-09-30T02:29:09.373 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:09.842 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:29:10.028 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:10.029 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-09-30T02:29:10.030 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:10.131 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse 2023-09-30T02:29:10.969 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:29:10.970 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:10.971 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 90 k 2023-09-30T02:29:10.971 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 264 k 2023-09-30T02:29:10.971 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:29:10.981 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:52 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:11.462 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 184 kB/s | 90 kB 00:00 2023-09-30T02:29:11.463 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:29:11.463 INFO:teuthology.orchestra.run.smithi157.stdout:Total 183 kB/s | 90 kB 00:00 2023-09-30T02:29:11.463 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:29:11.473 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:29:11.473 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:29:11.523 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:29:11.526 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:29:11.632 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:29:11.869 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:12.164 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:12.576 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:12.577 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:12.577 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-09-30T02:29:12.577 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 90 k 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 264 k 2023-09-30T02:29:12.578 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:29:12.809 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:12.809 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:12.809 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:29:12.809 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:29:12.809 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:12.809 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:12.985 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 222 kB/s | 90 kB 00:00 2023-09-30T02:29:12.986 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:29:12.986 INFO:teuthology.orchestra.run.smithi115.stdout:Total 221 kB/s | 90 kB 00:00 2023-09-30T02:29:12.986 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:29:12.996 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:29:12.996 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:29:13.011 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2023-09-30T02:29:13.045 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:29:13.048 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:29:13.177 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:29:13.446 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:13.728 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:13.874 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:02:55 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:14.307 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:14.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:29:14.308 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:29:14.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:14.308 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:14.506 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2023-09-30T02:29:15.363 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:57 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:15.464 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:15.464 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:15.464 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2023-09-30T02:29:15.464 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:15.464 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:29:15.465 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T02:29:15.465 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:15.465 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:29:15.465 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:15.465 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:29:15.465 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:15.466 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.5 M 2023-09-30T02:29:15.466 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 16 M 2023-09-30T02:29:15.466 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:29:16.069 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 5.7 MB/s | 3.5 MB 00:00 2023-09-30T02:29:16.070 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:29:16.070 INFO:teuthology.orchestra.run.smithi157.stdout:Total 5.7 MB/s | 3.5 MB 00:00 2023-09-30T02:29:16.070 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:29:16.082 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:29:16.082 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:29:16.146 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:29:16.149 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:29:16.590 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:29:16.845 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:16.918 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2023-09-30T02:29:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:16.918 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:29:16.918 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T02:29:16.919 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:29:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:29:16.919 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:16.919 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.5 M 2023-09-30T02:29:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 16 M 2023-09-30T02:29:16.920 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:29:17.298 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:17.840 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 3.8 MB/s | 3.5 MB 00:00 2023-09-30T02:29:17.841 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:29:17.841 INFO:teuthology.orchestra.run.smithi115.stdout:Total 3.7 MB/s | 3.5 MB 00:00 2023-09-30T02:29:17.841 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:29:17.853 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:29:17.853 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:29:17.915 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:29:17.918 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:29:17.945 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:17.945 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:17.945 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:29:17.945 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:29:17.945 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:17.945 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:18.124 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2023-09-30T02:29:18.350 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:29:18.579 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:18.993 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:03:00 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:18.998 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:19.614 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:29:19.614 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:29:19.615 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:19.615 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:19.792 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd 2023-09-30T02:29:20.590 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:20.591 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-09-30T02:29:20.592 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-09-30T02:29:20.592 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2023-09-30T02:29:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:20.592 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 173 k 2023-09-30T02:29:20.592 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 552 k 2023-09-30T02:29:20.593 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-09-30T02:29:20.653 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:03:02 ago on Sat 30 Sep 2023 02:26:18 AM UTC. 2023-09-30T02:29:20.888 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 585 kB/s | 173 kB 00:00 2023-09-30T02:29:20.889 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:29:20.889 INFO:teuthology.orchestra.run.smithi157.stdout:Total 583 kB/s | 173 kB 00:00 2023-09-30T02:29:20.889 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-09-30T02:29:20.900 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-09-30T02:29:20.900 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-09-30T02:29:20.949 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-09-30T02:29:20.952 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-09-30T02:29:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-09-30T02:29:21.284 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:22.173 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-09-30T02:29:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:29:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:22.173 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-09-30T02:29:22.233 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-09-30T02:29:22.233 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:22.233 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-09-30T02:29:22.233 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 173 k 2023-09-30T02:29:22.234 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 552 k 2023-09-30T02:29:22.235 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-09-30T02:29:22.352 DEBUG:teuthology.parallel:result is None 2023-09-30T02:29:22.426 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 903 kB/s | 173 kB 00:00 2023-09-30T02:29:22.427 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:29:22.427 INFO:teuthology.orchestra.run.smithi115.stdout:Total 897 kB/s | 173 kB 00:00 2023-09-30T02:29:22.427 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-09-30T02:29:22.438 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-09-30T02:29:22.438 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-09-30T02:29:22.487 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-09-30T02:29:22.490 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-09-30T02:29:22.610 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-09-30T02:29:22.802 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:23.108 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:23.698 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:29:23.699 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:23.699 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-09-30T02:29:23.699 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:29:23.699 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:23.699 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-09-30T02:29:23.911 DEBUG:teuthology.parallel:result is None 2023-09-30T02:29:23.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:29:24.132 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T02:29:24.154 INFO:teuthology.orchestra.run.smithi115.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T02:29:24.155 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T02:29:24.155 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T02:29:24.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:29:24.294 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T02:29:24.323 INFO:teuthology.orchestra.run.smithi157.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T02:29:24.323 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T02:29:24.323 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T02:29:24.323 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T02:29:24.324 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:24.324 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T02:29:24.352 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:24.353 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T02:29:24.396 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T02:29:24.397 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:24.397 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T02:29:24.424 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T02:29:24.491 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:24.491 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T02:29:24.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T02:29:24.594 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T02:29:24.595 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:24.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T02:29:24.621 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T02:29:24.688 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:24.689 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T02:29:24.719 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T02:29:24.789 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T02:29:24.789 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:24.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T02:29:24.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T02:29:24.883 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:24.883 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T02:29:24.913 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T02:29:24.982 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T02:29:25.049 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T02:29:25.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /var/log/ceph 2023-09-30T02:29:25.051 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2023-09-30T02:29:25.079 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T02:29:25.080 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T02:29:25.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T02:29:25.148 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T02:29:25.148 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T02:29:25.186 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T02:29:25.226 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T02:29:25.226 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T02:29:25.226 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7407271', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYOankZWKJky5kfFILmXO0eIT66frOaGpGYvbpK0S+mqX8fpoL6QLneZhLl/VEuLdRf2xx07HivIisYwtRluaw=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF8TKoW78maGXgk+0aMXgSg1b+cRx/KzgQ1eHyyhdluTQs7ZFD21iieBGxAfqEz43v1NGebJS9xtuog7nnLnJ/g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250', 'sudo ceph osd pool set cache min_read_recency_for_promote 2', 'sudo ceph osd pool set cache min_write_recency_for_promote 2']}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T02:29:25.227 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T02:29:25.253 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T02:29:25.273 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T02:29:25.311 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T02:29:25.340 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:25.340 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T02:29:25.394 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T02:29:25.394 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 391722 Links: 1 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 02:29:18.750878771 +0000 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 02:25:01.612788929 +0000 2023-09-30T02:29:25.452 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 02:25:01.612788929 +0000 2023-09-30T02:29:25.453 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T02:29:25.453 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T02:29:25.518 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T02:29:25.518 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T02:29:25.518 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000245047 s, 2.1 MB/s 2023-09-30T02:29:25.519 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T02:29:25.579 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 391788 Links: 1 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 02:29:18.750878771 +0000 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 02:25:02.133779437 +0000 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 02:25:02.133779437 +0000 2023-09-30T02:29:25.637 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T02:29:25.638 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T02:29:25.703 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T02:29:25.703 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T02:29:25.704 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000212107 s, 2.4 MB/s 2023-09-30T02:29:25.705 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T02:29:25.763 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2023-09-30T02:29:25.821 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 391832 Links: 1 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 02:29:18.750878771 +0000 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 02:25:02.643770145 +0000 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 02:25:02.643770145 +0000 2023-09-30T02:29:25.822 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T02:29:25.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T02:29:25.887 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T02:29:25.888 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T02:29:25.888 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000208689 s, 2.5 MB/s 2023-09-30T02:29:25.889 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T02:29:25.947 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 391923 Links: 1 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-09-30 02:29:18.750878771 +0000 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-09-30 02:25:03.155760817 +0000 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-09-30 02:25:03.155760817 +0000 2023-09-30T02:29:26.006 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-09-30T02:29:26.007 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T02:29:26.071 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-09-30T02:29:26.071 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-09-30T02:29:26.072 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000226984 s, 2.3 MB/s 2023-09-30T02:29:26.072 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T02:29:26.131 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:26.131 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:26.131 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T02:29:26.152 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T02:29:26.152 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2023-09-30T02:29:26.212 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T02:29:26.212 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:26.212 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 394333 Links: 1 2023-09-30T02:29:26.212 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:26.213 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:26.213 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 02:29:17.048335381 +0000 2023-09-30T02:29:26.213 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 02:25:11.397056494 +0000 2023-09-30T02:29:26.213 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 02:25:11.397056494 +0000 2023-09-30T02:29:26.213 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T02:29:26.213 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T02:29:26.280 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T02:29:26.280 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T02:29:26.280 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000118967 s, 4.3 MB/s 2023-09-30T02:29:26.281 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T02:29:26.342 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2023-09-30T02:29:26.400 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 396343 Links: 1 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 02:29:17.048335381 +0000 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 02:25:11.918047342 +0000 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 02:25:11.918047342 +0000 2023-09-30T02:29:26.401 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T02:29:26.402 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T02:29:26.468 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T02:29:26.468 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T02:29:26.468 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000121389 s, 4.2 MB/s 2023-09-30T02:29:26.469 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T02:29:26.528 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2023-09-30T02:29:26.586 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T02:29:26.586 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:26.586 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 396446 Links: 1 2023-09-30T02:29:26.586 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:26.587 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:26.587 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 02:29:17.048335381 +0000 2023-09-30T02:29:26.587 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 02:25:12.451037978 +0000 2023-09-30T02:29:26.587 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 02:25:12.451037978 +0000 2023-09-30T02:29:26.587 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T02:29:26.587 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T02:29:26.652 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T02:29:26.652 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T02:29:26.652 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.00011109 s, 4.6 MB/s 2023-09-30T02:29:26.653 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T02:29:26.712 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2023-09-30T02:29:26.770 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T02:29:26.770 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:29:26.770 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 395838 Links: 1 2023-09-30T02:29:26.770 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:29:26.771 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:29:26.771 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 02:29:17.048335381 +0000 2023-09-30T02:29:26.771 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 02:25:12.962029001 +0000 2023-09-30T02:29:26.771 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 02:25:12.962029001 +0000 2023-09-30T02:29:26.771 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T02:29:26.771 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T02:29:26.836 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T02:29:26.836 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T02:29:26.836 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 8.6313e-05 s, 5.9 MB/s 2023-09-30T02:29:26.837 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T02:29:26.896 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:26.896 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T02:29:26.896 INFO:tasks.ceph:Generating config... 2023-09-30T02:29:26.897 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T02:29:26.897 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T02:29:26.898 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] osd backoff on degraded = True 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] osd backoff on peering = True 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T02:29:26.899 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd map cache size = 1 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd max backfills = 6 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd max scrubs = 3 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-09-30T02:29:26.900 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-09-30T02:29:26.901 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T02:29:26.901 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2023-09-30T02:29:26.901 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2023-09-30T02:29:26.901 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T02:29:26.901 INFO:tasks.ceph:[global] ms type = async 2023-09-30T02:29:26.901 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2023-09-30T02:29:26.901 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2023-09-30T02:29:26.901 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-09-30T02:29:26.901 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2023-09-30T02:29:26.901 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T02:29:26.901 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T02:29:26.901 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T02:29:26.902 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T02:29:26.902 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T02:29:26.902 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T02:29:26.902 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T02:29:26.903 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T02:29:26.903 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T02:29:26.904 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T02:29:26.904 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T02:29:26.904 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T02:29:26.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T02:29:26.979 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T02:29:26.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T02:29:27.027 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T02:29:27.055 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.115'), ('mon.c', '[v2:172.21.15.115:3301,v1:172.21.15.115:6790]'), ('mon.b', '172.21.15.157')] 2023-09-30T02:29:27.056 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.115,[v2:172.21.15.115:3301,v1:172.21.15.115:6790],172.21.15.157', 'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T02:29:27.057 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:27.057 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T02:29:27.114 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.115 --addv c '[v2:172.21.15.115:3301,v1:172.21.15.115:6790]' --add b 172.21.15.157 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:27.199 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:27.199 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: generated fsid e8e31025-f333-40f5-9e8c-07777b1949b2 2023-09-30T02:29:27.199 INFO:teuthology.orchestra.run.smithi115.stdout:setting min_mon_release = octopus 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:epoch 0 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:fsid e8e31025-f333-40f5-9e8c-07777b1949b2 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:last_changed 2023-09-30T02:29:27.200592+0000 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:created 2023-09-30T02:29:27.200592+0000 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:min_mon_release 15 (octopus) 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:election_strategy: 1 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:2: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2023-09-30T02:29:27.200 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T02:29:27.201 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T02:29:27.220 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e8e31025-f333-40f5-9e8c-07777b1949b2... 2023-09-30T02:29:27.221 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T02:29:27.262 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T02:29:27.304 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2023-09-30T02:29:27.304 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.305 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2023-09-30T02:29:27.306 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2023-09-30T02:29:27.307 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.308 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T02:29:27.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.308 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2023-09-30T02:29:27.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.308 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2023-09-30T02:29:27.309 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.309 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2023-09-30T02:29:27.309 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2023-09-30T02:29:27.309 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2023-09-30T02:29:27.310 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = "172.21.15.115,[v2:172.21.15.115:3301,v1:172.21.15.115:6790],172.21.15.157" 2023-09-30T02:29:27.310 INFO:teuthology.orchestra.run.smithi115.stdout: mon election default strategy = 1 2023-09-30T02:29:27.310 INFO:teuthology.orchestra.run.smithi115.stdout: ms type = async 2023-09-30T02:29:27.310 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug inject dispatch delay duration = 0.1 2023-09-30T02:29:27.311 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug inject dispatch delay probability = 0.1 2023-09-30T02:29:27.311 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_min_size = 2 2023-09-30T02:29:27.311 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_size = 2 2023-09-30T02:29:27.311 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = e8e31025-f333-40f5-9e8c-07777b1949b2 2023-09-30T02:29:27.312 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.312 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2023-09-30T02:29:27.312 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2023-09-30T02:29:27.312 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.313 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2023-09-30T02:29:27.313 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 120 2023-09-30T02:29:27.314 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock profile = high_recovery_ops 2023-09-30T02:29:27.314 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.314 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2023-09-30T02:29:27.315 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2023-09-30T02:29:27.315 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.315 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2023-09-30T02:29:27.315 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = true 2023-09-30T02:29:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2023-09-30T02:29:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2023-09-30T02:29:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2023-09-30T02:29:27.316 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2023-09-30T02:29:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2023-09-30T02:29:27.317 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2023-09-30T02:29:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2023-09-30T02:29:27.318 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2023-09-30T02:29:27.319 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.319 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2023-09-30T02:29:27.319 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2023-09-30T02:29:27.319 INFO:teuthology.orchestra.run.smithi115.stdout: debug monc = 20 2023-09-30T02:29:27.319 INFO:teuthology.orchestra.run.smithi115.stdout: bluefs allocator = hybrid 2023-09-30T02:29:27.320 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore allocator = hybrid 2023-09-30T02:29:27.320 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2023-09-30T02:29:27.320 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2023-09-30T02:29:27.320 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore zero block detection = True 2023-09-30T02:29:27.320 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 20 2023-09-30T02:29:27.321 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 20 2023-09-30T02:29:27.321 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T02:29:27.321 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2023-09-30T02:29:27.321 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 10 2023-09-30T02:29:27.321 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T02:29:27.322 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2023-09-30T02:29:27.322 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2023-09-30T02:29:27.322 INFO:teuthology.orchestra.run.smithi115.stdout: osd backoff on degraded = True 2023-09-30T02:29:27.322 INFO:teuthology.orchestra.run.smithi115.stdout: osd backoff on peering = True 2023-09-30T02:29:27.322 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify cached snaps = True 2023-09-30T02:29:27.323 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify missing on start = True 2023-09-30T02:29:27.323 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2023-09-30T02:29:27.324 INFO:teuthology.orchestra.run.smithi115.stdout: osd inject bad map crc probability = 0.1 2023-09-30T02:29:27.324 INFO:teuthology.orchestra.run.smithi115.stdout: osd map cache size = 1 2023-09-30T02:29:27.324 INFO:teuthology.orchestra.run.smithi115.stdout: osd max backfills = 6 2023-09-30T02:29:27.324 INFO:teuthology.orchestra.run.smithi115.stdout: osd max markdown count = 1000 2023-09-30T02:29:27.325 INFO:teuthology.orchestra.run.smithi115.stdout: osd max scrubs = 3 2023-09-30T02:29:27.325 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock override recovery settings = True 2023-09-30T02:29:27.325 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2023-09-30T02:29:27.325 INFO:teuthology.orchestra.run.smithi115.stdout: osd op queue = debug_random 2023-09-30T02:29:27.325 INFO:teuthology.orchestra.run.smithi115.stdout: osd op queue cut off = debug_random 2023-09-30T02:29:27.325 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub during recovery = False 2023-09-30T02:29:27.326 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub min interval = 60 2023-09-30T02:29:27.326 INFO:teuthology.orchestra.run.smithi115.stdout: osd shutdown pgref assert = True 2023-09-30T02:29:27.326 INFO:teuthology.orchestra.run.smithi115.stdout: osd_max_pg_log_entries = 3000 2023-09-30T02:29:27.326 INFO:teuthology.orchestra.run.smithi115.stdout: osd_min_pg_log_entries = 3000 2023-09-30T02:29:27.326 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.326 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2023-09-30T02:29:27.327 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T02:29:27.327 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2023-09-30T02:29:27.327 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2023-09-30T02:29:27.327 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2023-09-30T02:29:27.327 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:29:27.327 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:29:27.328 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2023-09-30T02:29:27.328 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.328 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2023-09-30T02:29:27.328 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-09-30T02:29:27.328 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2023-09-30T02:29:27.328 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2023-09-30T02:29:27.329 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2023-09-30T02:29:27.329 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2023-09-30T02:29:27.329 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2023-09-30T02:29:27.329 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:29:27.329 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2023-09-30T02:29:27.329 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2023-09-30T02:29:27.330 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:29:27.330 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.330 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T02:29:27.330 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T02:29:27.331 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T02:29:27.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.331 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T02:29:27.331 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T02:29:27.331 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T02:29:27.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.332 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2023-09-30T02:29:27.332 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2023-09-30T02:29:27.332 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.332 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2023-09-30T02:29:27.333 INFO:teuthology.orchestra.run.smithi115.stdout: mon min osdmap epochs = 50 2023-09-30T02:29:27.333 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune interval = 2 2023-09-30T02:29:27.333 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune min = 15 2023-09-30T02:29:27.333 INFO:teuthology.orchestra.run.smithi115.stdout: mon osdmap full prune txsize = 2 2023-09-30T02:29:27.333 INFO:teuthology.orchestra.run.smithi115.stdout: mon scrub interval = 300 2023-09-30T02:29:27.333 INFO:teuthology.orchestra.run.smithi115.stdout: paxos service trim min = 10 2023-09-30T02:29:27.334 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:29:27.334 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2023-09-30T02:29:27.334 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2023-09-30T02:29:27.334 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2023-09-30T02:29:27.335 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2023-09-30T02:29:27.335 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T02:29:27.335 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T02:29:27.335 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2023-09-30T02:29:27.335 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.c] 2023-09-30T02:29:27.335 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.b] 2023-09-30T02:29:27.337 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2023-09-30T02:29:27.337 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2023-09-30T02:29:27.338 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T02:29:27.338 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2023-09-30T02:29:27.338 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.338 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2023-09-30T02:29:27.338 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.338 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2023-09-30T02:29:27.339 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2023-09-30T02:29:27.340 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2023-09-30T02:29:27.341 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.115,[v2:172.21.15.115:3301,v1:172.21.15.115:6790],172.21.15.157" 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: mon election default strategy = 1 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: ms type = async 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug inject dispatch delay duration = 0.1 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug inject dispatch delay probability = 0.1 2023-09-30T02:29:27.342 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_min_size = 2 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_size = 2 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = e8e31025-f333-40f5-9e8c-07777b1949b2 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 120 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2023-09-30T02:29:27.343 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.344 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2023-09-30T02:29:27.344 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2023-09-30T02:29:27.344 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2023-09-30T02:29:27.344 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.344 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2023-09-30T02:29:27.344 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.345 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: bluefs allocator = hybrid 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore allocator = hybrid 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore zero block detection = True 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T02:29:27.346 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd backoff on degraded = True 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd backoff on peering = True 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify cached snaps = True 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify missing on start = True 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd inject bad map crc probability = 0.1 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd map cache size = 1 2023-09-30T02:29:27.347 INFO:teuthology.orchestra.run.smithi157.stdout: osd max backfills = 6 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd max scrubs = 3 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock override recovery settings = True 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue = debug_random 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue cut off = debug_random 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub during recovery = False 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub min interval = 60 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_pg_log_entries = 3000 2023-09-30T02:29:27.348 INFO:teuthology.orchestra.run.smithi157.stdout: osd_min_pg_log_entries = 3000 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2023-09-30T02:29:27.349 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T02:29:27.350 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: mon min osdmap epochs = 50 2023-09-30T02:29:27.351 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune interval = 2 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune min = 15 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune txsize = 2 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: mon scrub interval = 300 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: paxos service trim min = 10 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T02:29:27.352 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2023-09-30T02:29:27.353 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2023-09-30T02:29:27.353 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2023-09-30T02:29:27.353 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T02:29:27.353 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T02:29:27.431 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T02:29:27.431 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:27.432 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T02:29:27.450 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:27.450 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T02:29:27.507 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:29:27.507 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:27.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T02:29:27.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T02:29:27.584 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:27.584 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:27.642 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:29:27.643 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:27.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T02:29:27.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T02:29:27.681 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:27.681 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:27.739 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T02:29:27.739 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T02:29:27.739 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T02:29:27.789 INFO:teuthology.orchestra.run.smithi115.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T02:29:27.792 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T02:29:27.873 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T02:29:27.876 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T02:29:27.876 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T02:29:27.876 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T02:29:27.917 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T02:29:27.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T02:29:27.969 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T02:29:27.982 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T02:29:27.982 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T02:29:27.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T02:29:28.009 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:28.009 INFO:tasks.ceph:role: osd.0 2023-09-30T02:29:28.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:29:28.009 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T02:29:28.083 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:28.084 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:28.519 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T02:29:28.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T02:29:28.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T02:29:28.703 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T02:29:28.735 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T02:29:28.803 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:28.803 INFO:tasks.ceph:role: osd.1 2023-09-30T02:29:28.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:29:28.803 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T02:29:28.877 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:28.877 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:28.877 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:28.877 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:28.878 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:28.878 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:28.878 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:28.878 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:28.878 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:28.878 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:29.303 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T02:29:29.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T02:29:29.305 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T02:29:29.411 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T02:29:29.442 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T02:29:29.510 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:29.510 INFO:tasks.ceph:role: osd.2 2023-09-30T02:29:29.510 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:29:29.511 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T02:29:29.585 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:29.585 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:29.585 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:29.585 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:29.586 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:29.586 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:29.586 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:29.586 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:29.586 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:29.586 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:30.031 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T02:29:30.033 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T02:29:30.033 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T02:29:30.211 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T02:29:30.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T02:29:30.314 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:30.314 INFO:tasks.ceph:role: osd.3 2023-09-30T02:29:30.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:29:30.314 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T02:29:30.388 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:30.389 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:30.834 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-09-30T02:29:30.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-09-30T02:29:30.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T02:29:31.023 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T02:29:31.057 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:31.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.181+0000 7f1def7c7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T02:29:31.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.182+0000 7f1def7c7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T02:29:31.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.182+0000 7f1def7c7540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T02:29:31.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.182+0000 7f1def7c7540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T02:29:31.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.182+0000 7f1def7c7540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T02:29:31.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.182+0000 7f1def7c7540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T02:29:31.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:31.706+0000 7f1def7c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:32.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:32.472+0000 7f1def7c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:33.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:33.237+0000 7f1def7c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:34.035 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T02:29:34.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:34.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.152+0000 7f9aad8b7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T02:29:34.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.152+0000 7f9aad8b7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T02:29:34.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.152+0000 7f9aad8b7540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T02:29:34.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.152+0000 7f9aad8b7540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T02:29:34.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.152+0000 7f9aad8b7540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T02:29:34.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.153+0000 7f9aad8b7540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T02:29:34.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:34.679+0000 7f9aad8b7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:35.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:35.445+0000 7f9aad8b7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:36.209 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:36.210+0000 7f9aad8b7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:37.002 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T02:29:37.033 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:37.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.121+0000 7f1c3c704540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T02:29:37.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.122+0000 7f1c3c704540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T02:29:37.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.122+0000 7f1c3c704540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T02:29:37.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.122+0000 7f1c3c704540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T02:29:37.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.122+0000 7f1c3c704540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T02:29:37.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.122+0000 7f1c3c704540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T02:29:37.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:37.647+0000 7f1c3c704540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:38.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:38.411+0000 7f1c3c704540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:39.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:39.173+0000 7f1c3c704540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:39.970 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T02:29:40.002 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:40.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.094+0000 7f4d6fe30540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T02:29:40.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.095+0000 7f4d6fe30540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T02:29:40.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.095+0000 7f4d6fe30540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T02:29:40.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.095+0000 7f4d6fe30540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T02:29:40.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.095+0000 7f4d6fe30540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T02:29:40.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.095+0000 7f4d6fe30540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T02:29:40.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:40.620+0000 7f4d6fe30540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:41.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:41.382+0000 7f4d6fe30540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:42.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-09-30T02:29:42.148+0000 7f4d6fe30540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:42.944 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T02:29:42.977 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T02:29:43.012 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:43.012 INFO:tasks.ceph:role: osd.4 2023-09-30T02:29:43.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:29:43.012 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:43.083 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:43.084 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:43.106 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T02:29:43.107 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T02:29:43.107 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T02:29:43.190 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T02:29:43.263 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T02:29:43.331 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:43.331 INFO:tasks.ceph:role: osd.5 2023-09-30T02:29:43.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:29:43.331 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:43.398 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:43.399 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:43.399 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:43.399 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:43.420 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T02:29:43.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T02:29:43.422 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T02:29:43.502 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T02:29:43.573 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T02:29:43.641 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:43.641 INFO:tasks.ceph:role: osd.6 2023-09-30T02:29:43.641 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:29:43.642 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T02:29:43.708 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:43.708 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:43.708 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:43.708 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:43.708 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:43.708 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:43.709 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:43.709 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:43.709 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:43.709 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:43.730 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T02:29:43.732 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T02:29:43.732 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T02:29:43.812 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T02:29:43.883 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T02:29:43.950 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:29:43.950 INFO:tasks.ceph:role: osd.7 2023-09-30T02:29:43.950 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:29:43.950 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:29:44.017 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:29:44.018 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:29:44.018 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:29:44.039 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2023-09-30T02:29:44.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T02:29:44.041 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T02:29:44.121 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T02:29:44.192 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:44.285 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.284+0000 7f497b934540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T02:29:44.285 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.284+0000 7f497b934540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T02:29:44.285 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.284+0000 7f497b934540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T02:29:44.285 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.284+0000 7f497b934540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T02:29:44.285 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.284+0000 7f497b934540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T02:29:44.286 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.285+0000 7f497b934540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T02:29:44.805 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:44.805+0000 7f497b934540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:45.567 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:45.567+0000 7f497b934540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:46.330 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:46.330+0000 7f497b934540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:47.125 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T02:29:47.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:47.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.246+0000 7f423a0e4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T02:29:47.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.246+0000 7f423a0e4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T02:29:47.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.246+0000 7f423a0e4540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T02:29:47.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.246+0000 7f423a0e4540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T02:29:47.248 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.246+0000 7f423a0e4540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T02:29:47.248 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.247+0000 7f423a0e4540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T02:29:47.770 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:47.769+0000 7f423a0e4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:48.538 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:48.537+0000 7f423a0e4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:49.299 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:49.299+0000 7f423a0e4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:50.096 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T02:29:50.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:50.218 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.218+0000 7f2bc0e4c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T02:29:50.218 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.218+0000 7f2bc0e4c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T02:29:50.219 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.218+0000 7f2bc0e4c540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T02:29:50.219 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.218+0000 7f2bc0e4c540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T02:29:50.219 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.218+0000 7f2bc0e4c540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T02:29:50.219 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.218+0000 7f2bc0e4c540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T02:29:50.741 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:50.741+0000 7f2bc0e4c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:51.506 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:51.506+0000 7f2bc0e4c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:52.271 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:52.270+0000 7f2bc0e4c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:53.065 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T02:29:53.098 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:53.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.186+0000 7f3bcdc12540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T02:29:53.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.186+0000 7f3bcdc12540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T02:29:53.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.186+0000 7f3bcdc12540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T02:29:53.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.186+0000 7f3bcdc12540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T02:29:53.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.186+0000 7f3bcdc12540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T02:29:53.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.186+0000 7f3bcdc12540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T02:29:53.715 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:53.714+0000 7f3bcdc12540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:54.477 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:54.477+0000 7f3bcdc12540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:55.240 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T02:29:55.240+0000 7f3bcdc12540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:29:56.036 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T02:29:56.068 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T02:29:56.069 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:56.069 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T02:29:56.103 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:56.103 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T02:29:56.175 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:56.175 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T02:29:56.246 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:56.247 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T02:29:56.317 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:56.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T02:29:56.389 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:56.389 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T02:29:56.422 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:56.422 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T02:29:56.493 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:56.494 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T02:29:56.565 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:56.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T02:29:56.638 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:56.638 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T02:29:56.709 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:29:56.709 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T02:29:56.731 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:29:56.731 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T02:29:56.769 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T02:29:56.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T02:29:56.773 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.y] 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAHiBdlmZ8kLxAAda9d0+MF/pkaBtHetRrVpQ== 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.0] 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQALiBdlpvfkChAAFuy34v8sLwksLPIQ+aBI9A== 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.1] 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAOiBdlzKYnCRAAGz47VSSHnpsp2yGINaf0Zw== 2023-09-30T02:29:56.812 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.2] 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQARiBdlM69QBxAAxAzZ+yqk/XeR0aFtlkGlZg== 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.3] 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAUiBdlDpizBRAANXfTugu4aAnZL0kl6YHN2Q== 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.x] 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAHiBdlhZclNBAAnlOaDKBEDzP8fQGx+a72Sg== 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.4] 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAYiBdlkYoLERAAIRBLuA0pChJCYb1OboTFEw== 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.5] 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAbiBdlz+zGDhAAtVMHTbbXOMTjJW9lFrxdMw== 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.6] 2023-09-30T02:29:56.813 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAeiBdlfmESDRAAMC4XBNeGgSOgMH9r9aNdsg== 2023-09-30T02:29:56.814 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.7] 2023-09-30T02:29:56.814 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAhiBdlxgUpCxAAE3Zl70LIr+mdHXxd5tCKAg== 2023-09-30T02:29:56.814 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2023-09-30T02:29:56.814 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAHiBdlFqe8NhAAvr8S+6J7i65BhK/KQbghsQ== 2023-09-30T02:29:56.814 INFO:teuthology.orchestra.run.smithi115.stdout:[client.1] 2023-09-30T02:29:56.814 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQAHiBdlZkneORAAwfvTRXUnQEevu0QYfr/SdQ== 2023-09-30T02:29:56.837 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAHiBdlmZ8kLxAAda9d0+MF/pkaBtHetRrVpQ== 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQALiBdlpvfkChAAFuy34v8sLwksLPIQ+aBI9A== 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAOiBdlzKYnCRAAGz47VSSHnpsp2yGINaf0Zw== 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQARiBdlM69QBxAAxAzZ+yqk/XeR0aFtlkGlZg== 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAUiBdlDpizBRAANXfTugu4aAnZL0kl6YHN2Q== 2023-09-30T02:29:56.838 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAHiBdlhZclNBAAnlOaDKBEDzP8fQGx+a72Sg== 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAYiBdlkYoLERAAIRBLuA0pChJCYb1OboTFEw== 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAbiBdlz+zGDhAAtVMHTbbXOMTjJW9lFrxdMw== 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAeiBdlfmESDRAAMC4XBNeGgSOgMH9r9aNdsg== 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAhiBdlxgUpCxAAE3Zl70LIr+mdHXxd5tCKAg== 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAHiBdlFqe8NhAAvr8S+6J7i65BhK/KQbghsQ== 2023-09-30T02:29:56.839 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2023-09-30T02:29:56.840 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQAHiBdlZkneORAAwfvTRXUnQEevu0QYfr/SdQ== 2023-09-30T02:29:56.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:29:56.857 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:29:56.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:56.936 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:56.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:56.989 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.040 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.090 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.092 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:29:57.144 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:29:57.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.196 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.246 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.248 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.298 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.300 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.352 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:29:57.402 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:29:57.404 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:29:57.457 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:29:57.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:29:57.512 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T02:29:57.512 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T02:29:57.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T02:29:57.635 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T02:29:57.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T02:29:57.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T02:29:57.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T02:29:57.851 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T02:29:57.880 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T02:29:57.977 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T02:29:58.004 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:58.006 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:29:58.062 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T02:29:58.062 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T02:29:58.062 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T02:29:58.064 INFO:tasks.ceph.mon.a:Started 2023-09-30T02:29:58.064 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T02:29:58.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T02:29:58.066 INFO:tasks.ceph.mon.c:Started 2023-09-30T02:29:58.066 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T02:29:58.067 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T02:29:58.104 INFO:tasks.ceph.mon.b:Started 2023-09-30T02:29:58.105 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T02:29:58.105 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T02:29:58.105 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T02:29:58.107 INFO:tasks.ceph.mgr.y:Started 2023-09-30T02:29:58.107 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T02:29:58.107 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T02:29:58.109 INFO:tasks.ceph.mgr.x:Started 2023-09-30T02:29:58.109 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T02:29:58.109 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T02:30:03.516 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2023-09-30T02:30:03.528 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T02:30:03.528 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:30:03.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T02:30:03.556 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:30:03.557 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T02:30:03.561 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:03.561+0000 7fbcc26611c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T02:30:03.572 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:03.572+0000 7f7543f731c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T02:30:03.600 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:30:03.600 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T02:30:03.675 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-09-30T02:30:03.675 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T02:30:03.726 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:03.726+0000 7fbcc26611c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T02:30:03.749 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:30:03.749 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T02:30:03.750 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:03.749+0000 7f7543f731c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T02:30:03.778 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:30:03.778 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T02:30:03.846 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:30:03.847 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T02:30:03.914 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T02:30:03.914 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T02:30:03.981 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new a0eead8b-1085-45e3-82b6-6fdbbeda801a 0 2023-09-30T02:30:04.101 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:04.101+0000 7f7543f731c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T02:30:04.106 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:04.106+0000 7fbcc26611c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T02:30:04.235 INFO:teuthology.orchestra.run.smithi157.stdout:0 2023-09-30T02:30:04.248 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new a459ef89-74f1-472d-a093-6020937471e6 1 2023-09-30T02:30:04.464 INFO:teuthology.orchestra.run.smithi157.stdout:1 2023-09-30T02:30:04.477 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4 2 2023-09-30T02:30:04.592 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:04.592+0000 7f7543f731c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T02:30:04.636 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:04.636+0000 7fbcc26611c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T02:30:04.690 INFO:teuthology.orchestra.run.smithi157.stdout:2 2023-09-30T02:30:04.702 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 85677263-5295-4c4b-bb83-1b01ba58356c 3 2023-09-30T02:30:04.730 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:04.730+0000 7f7543f731c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T02:30:04.781 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:04.780+0000 7fbcc26611c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T02:30:04.916 INFO:teuthology.orchestra.run.smithi157.stdout:3 2023-09-30T02:30:04.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 2cce4678-d9e6-411e-8fe1-3b8aef44ab18 4 2023-09-30T02:30:05.063 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:05.064+0000 7f7543f731c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T02:30:05.133 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:05.133+0000 7fbcc26611c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T02:30:05.139 INFO:teuthology.orchestra.run.smithi157.stdout:4 2023-09-30T02:30:05.152 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 9bfeeafc-a943-41c9-8552-b83a7945a936 5 2023-09-30T02:30:05.367 INFO:teuthology.orchestra.run.smithi157.stdout:5 2023-09-30T02:30:05.380 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 8e859338-35cf-4c43-b0d0-1fede34770d8 6 2023-09-30T02:30:05.592 INFO:teuthology.orchestra.run.smithi157.stdout:6 2023-09-30T02:30:05.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new ff1fa622-2c14-4469-ae34-10d233bfabd0 7 2023-09-30T02:30:05.781 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:05.781+0000 7f7543f731c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T02:30:05.815 INFO:teuthology.orchestra.run.smithi157.stdout:7 2023-09-30T02:30:05.827 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T02:30:05.828 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T02:30:05.830 INFO:tasks.ceph.osd.0:Started 2023-09-30T02:30:05.830 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T02:30:05.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T02:30:05.832 INFO:tasks.ceph.osd.1:Started 2023-09-30T02:30:05.832 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T02:30:05.832 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T02:30:05.834 INFO:tasks.ceph.osd.2:Started 2023-09-30T02:30:05.834 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T02:30:05.834 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T02:30:05.836 INFO:tasks.ceph.osd.3:Started 2023-09-30T02:30:05.836 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T02:30:05.836 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T02:30:05.838 INFO:tasks.ceph.osd.4:Started 2023-09-30T02:30:05.838 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T02:30:05.839 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T02:30:05.841 INFO:tasks.ceph.osd.5:Started 2023-09-30T02:30:05.841 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T02:30:05.841 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-30T02:30:05.843 INFO:tasks.ceph.osd.6:Started 2023-09-30T02:30:05.843 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T02:30:05.843 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-30T02:30:05.845 INFO:tasks.ceph.osd.7:Started 2023-09-30T02:30:05.845 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:30:05.863 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:05.863+0000 7f7543f731c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T02:30:05.909 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:05.909+0000 7fbcc26611c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T02:30:05.987 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:05.987+0000 7fbcc26611c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T02:30:06.017 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.017+0000 7f7543f731c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T02:30:06.075 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:06.075 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":10,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:30:05.813739+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:30:06.087 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T02:30:06.087 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T02:30:06.088 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.087+0000 7f7543f731c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T02:30:06.120 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.119+0000 7fbcc26611c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T02:30:06.187 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.187+0000 7fbcc26611c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T02:30:06.267 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.266+0000 7f7543f731c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T02:30:06.342 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.342+0000 7f7543f731c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T02:30:06.367 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.367+0000 7fbcc26611c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T02:30:06.417 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.417+0000 7f7543f731c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T02:30:06.437 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T02:30:06.437+0000 7f5e88905540 -1 Falling back to public interface 2023-09-30T02:30:06.445 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.445+0000 7fbcc26611c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T02:30:06.445 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:30:06.445+0000 7f1b56867540 -1 Falling back to public interface 2023-09-30T02:30:06.449 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T02:30:06.449+0000 7f5ec5d3a540 -1 Falling back to public interface 2023-09-30T02:30:06.450 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T02:30:06.450+0000 7f7e54f68540 -1 Falling back to public interface 2023-09-30T02:30:06.457 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:30:06.457+0000 7f352841f540 -1 Falling back to public interface 2023-09-30T02:30:06.458 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:30:06.458+0000 7fa7e7c6b540 -1 Falling back to public interface 2023-09-30T02:30:06.459 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:30:06.459+0000 7fa7e2077700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6823/0: (98) Address already in use 2023-09-30T02:30:06.459 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:30:06.460+0000 7f352282b700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6825/0: (98) Address already in use 2023-09-30T02:30:06.464 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:30:06.464+0000 7ff38f4c4540 -1 Falling back to public interface 2023-09-30T02:30:06.468 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:30:06.468+0000 7f3cfa503540 -1 Falling back to public interface 2023-09-30T02:30:06.507 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.507+0000 7f7543f731c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T02:30:06.529 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.529+0000 7fbcc26611c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T02:30:06.623 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.623+0000 7fbcc26611c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T02:30:06.682 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.682+0000 7f7543f731c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T02:30:06.762 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:06.763+0000 7f7543f731c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T02:30:06.802 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.801+0000 7fbcc26611c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T02:30:06.881 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:06.880+0000 7fbcc26611c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T02:30:06.891 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:30:07.049 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:07.049+0000 7f7543f731c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T02:30:07.078 INFO:teuthology.misc.health.smithi115.stdout: 2023-09-30T02:30:07.079 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":10,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:30:05.813739+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:30:07.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:30:07.120 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:07.121+0000 7f7543f731c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T02:30:07.169 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:07.168+0000 7fbcc26611c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T02:30:07.238 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:07.237+0000 7fbcc26611c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T02:30:07.626 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:07.626+0000 7f7543f731c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T02:30:07.746 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:07.746+0000 7fbcc26611c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T02:30:07.936 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:07.935+0000 7f7543f731c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T02:30:08.062 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:08.062+0000 7fbcc26611c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T02:30:08.076 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.075+0000 7f7543f731c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:30:08.205 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:08.205+0000 7fbcc26611c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:30:08.432 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.431+0000 7f7543f731c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T02:30:08.559 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:08.559+0000 7fbcc26611c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T02:30:08.572 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.571+0000 7f7543f731c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:30:08.704 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:08.703+0000 7fbcc26611c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:30:08.722 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.721+0000 7f7543f731c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T02:30:08.821 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.820+0000 7f7543f731c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T02:30:08.854 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:08.854+0000 7fbcc26611c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T02:30:08.863 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.862+0000 7f74d341d700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:30:08.863 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.862+0000 7f74d341d700 -1 pg_autoscaler.serve: 2023-09-30T02:30:08.863 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:08.862+0000 7f74d341d700 -1 Traceback (most recent call last): 2023-09-30T02:30:08.864 INFO:tasks.ceph.mgr.y.smithi115.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T02:30:08.864 INFO:tasks.ceph.mgr.y.smithi115.stderr: self._maybe_adjust() 2023-09-30T02:30:08.864 INFO:tasks.ceph.mgr.y.smithi115.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T02:30:08.864 INFO:tasks.ceph.mgr.y.smithi115.stderr: if self.noautoscale: 2023-09-30T02:30:08.864 INFO:tasks.ceph.mgr.y.smithi115.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:30:08.864 INFO:tasks.ceph.mgr.y.smithi115.stderr: 2023-09-30T02:30:08.965 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T02:30:08.965+0000 7fbcc26611c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T02:30:09.844 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:09.843+0000 7f7508657700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T02:30:09.844 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-09-30T02:30:09.844+0000 7f7508657700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T02:30:09.917 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:30:09.916+0000 7f1b56867540 -1 osd.0 0 log_to_monitors true 2023-09-30T02:30:09.962 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:30:09.961+0000 7fa7e7c6b540 -1 osd.2 0 log_to_monitors true 2023-09-30T02:30:09.965 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:30:09.965+0000 7ff38f4c4540 -1 osd.5 0 log_to_monitors true 2023-09-30T02:30:09.972 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T02:30:09.972+0000 7f5ec5d3a540 -1 osd.3 0 log_to_monitors true 2023-09-30T02:30:09.979 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T02:30:09.979+0000 7f5e88905540 -1 osd.7 0 log_to_monitors true 2023-09-30T02:30:09.984 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:30:09.984+0000 7f3cfa503540 -1 osd.6 0 log_to_monitors true 2023-09-30T02:30:09.987 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T02:30:09.987+0000 7f7e54f68540 -1 osd.4 0 log_to_monitors true 2023-09-30T02:30:09.994 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:30:09.993+0000 7f352841f540 -1 osd.1 0 log_to_monitors true 2023-09-30T02:30:11.951 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:30:11.950+0000 7fa7de870700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T02:30:11.951 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:30:11.950+0000 7f1b4d46c700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T02:30:11.952 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T02:30:11.951+0000 7f7e4bb6d700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T02:30:11.952 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:30:11.951+0000 7f3cf1108700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T02:30:11.952 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:30:11.951+0000 7ff3860c9700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T02:30:11.952 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T02:30:11.952+0000 7f5e80d0d700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T02:30:11.953 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T02:30:11.951+0000 7f5ebc93f700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T02:30:11.953 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:30:11.952+0000 7f3520827700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T02:30:11.958 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T02:30:11.957+0000 7f7e45165700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.958 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:30:11.957+0000 7ff37eec0700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.958 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T02:30:11.958+0000 7f5e78b02700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.958 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:30:11.958+0000 7f3ce9eff700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.960 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:30:11.959+0000 7f1b46a64700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.960 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:30:11.959+0000 7fa7d7667700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.961 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T02:30:11.959+0000 7f5eb5f37700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:11.963 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:30:11.962+0000 7f3517e1b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:30:12.954 INFO:tasks.ceph.mon.a.smithi115.stderr:2023-09-30T02:30:12.953+0000 7fbeb16da700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T02:30:13.897 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:30:14.214 INFO:teuthology.misc.health.smithi115.stdout: 2023-09-30T02:30:14.214 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":14,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:30:13.863889+0000","last_up_change":"2023-09-30T02:30:12.852491+0000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:30:12.888959+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115900},{"type":"v1","addr":"172.21.15.115:6801","nonce":115900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115900},{"type":"v1","addr":"172.21.15.115:6803","nonce":115900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115900},{"type":"v1","addr":"172.21.15.115:6807","nonce":115900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115900},{"type":"v1","addr":"172.21.15.115:6805","nonce":115900}]},"public_addr":"172.21.15.115:6801/115900","cluster_addr":"172.21.15.115:6803/115900","heartbeat_back_addr":"172.21.15.115:6807/115900","heartbeat_front_addr":"172.21.15.115:6805/115900","state":["exists","up"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":115902},{"type":"v1","addr":"172.21.15.115:6817","nonce":115902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":115902},{"type":"v1","addr":"172.21.15.115:6820","nonce":115902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":115902},{"type":"v1","addr":"172.21.15.115:6828","nonce":115902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":115902},{"type":"v1","addr":"172.21.15.115:6823","nonce":115902}]},"public_addr":"172.21.15.115:6817/115902","cluster_addr":"172.21.15.115:6820/115902","heartbeat_back_addr":"172.21.15.115:6828/115902","heartbeat_front_addr":"172.21.15.115:6823/115902","state":["exists","up"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6819","nonce":115903},{"type":"v1","addr":"172.21.15.115:6821","nonce":115903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":115903},{"type":"v1","addr":"172.21.15.115:6825","nonce":115903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":115903},{"type":"v1","addr":"172.21.15.115:6831","nonce":115903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6827","nonce":115903},{"type":"v1","addr":"172.21.15.115:6829","nonce":115903}]},"public_addr":"172.21.15.115:6821/115903","cluster_addr":"172.21.15.115:6825/115903","heartbeat_back_addr":"172.21.15.115:6831/115903","heartbeat_front_addr":"172.21.15.115:6829/115903","state":["exists","up"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115901},{"type":"v1","addr":"172.21.15.115:6809","nonce":115901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115901},{"type":"v1","addr":"172.21.15.115:6811","nonce":115901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115901},{"type":"v1","addr":"172.21.15.115:6815","nonce":115901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115901},{"type":"v1","addr":"172.21.15.115:6813","nonce":115901}]},"public_addr":"172.21.15.115:6809/115901","cluster_addr":"172.21.15.115:6811/115901","heartbeat_back_addr":"172.21.15.115:6815/115901","heartbeat_front_addr":"172.21.15.115:6813/115901","state":["exists","up"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115856},{"type":"v1","addr":"172.21.15.157:6809","nonce":115856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115856},{"type":"v1","addr":"172.21.15.157:6811","nonce":115856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115856},{"type":"v1","addr":"172.21.15.157:6815","nonce":115856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115856},{"type":"v1","addr":"172.21.15.157:6813","nonce":115856}]},"public_addr":"172.21.15.157:6809/115856","cluster_addr":"172.21.15.157:6811/115856","heartbeat_back_addr":"172.21.15.157:6815/115856","heartbeat_front_addr":"172.21.15.157:6813/115856","state":["exists","up"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":115886},{"type":"v1","addr":"172.21.15.157:6817","nonce":115886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":115886},{"type":"v1","addr":"172.21.15.157:6819","nonce":115886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":115886},{"type":"v1","addr":"172.21.15.157:6823","nonce":115886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":115886},{"type":"v1","addr":"172.21.15.157:6821","nonce":115886}]},"public_addr":"172.21.15.157:6817/115886","cluster_addr":"172.21.15.157:6819/115886","heartbeat_back_addr":"172.21.15.157:6823/115886","heartbeat_front_addr":"172.21.15.157:6821/115886","state":["exists","up"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":115887},{"type":"v1","addr":"172.21.15.157:6825","nonce":115887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":115887},{"type":"v1","addr":"172.21.15.157:6827","nonce":115887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":115887},{"type":"v1","addr":"172.21.15.157:6831","nonce":115887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":115887},{"type":"v1","addr":"172.21.15.157:6829","nonce":115887}]},"public_addr":"172.21.15.157:6825/115887","cluster_addr":"172.21.15.157:6827/115887","heartbeat_back_addr":"172.21.15.157:6831/115887","heartbeat_front_addr":"172.21.15.157:6829/115887","state":["exists","up"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115857},{"type":"v1","addr":"172.21.15.157:6801","nonce":115857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115857},{"type":"v1","addr":"172.21.15.157:6803","nonce":115857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115857},{"type":"v1","addr":"172.21.15.157:6807","nonce":115857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115857},{"type":"v1","addr":"172.21.15.157:6805","nonce":115857}]},"public_addr":"172.21.15.157:6801/115857","cluster_addr":"172.21.15.157:6803/115857","heartbeat_back_addr":"172.21.15.157:6807/115857","heartbeat_front_addr":"172.21.15.157:6805/115857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.966788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.953004+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.996123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.023275+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.014226+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.941302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.947814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.006181+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:30:14.227 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T02:30:14.227 INFO:tasks.ceph:Creating RBD pool 2023-09-30T02:30:14.228 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T02:30:14.879 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'rbd' created 2023-09-30T02:30:14.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T02:30:15.881 INFO:teuthology.orchestra.run.smithi115.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T02:30:15.893 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T02:30:15.893 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T02:30:15.893 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T02:30:15.894 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T02:30:15.894 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T02:30:16.265 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:16.276 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":4,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":115600},{"type":"v1","addr":"172.21.15.115:6833","nonce":115600}]},"active_addr":"172.21.15.115:6833/115600","active_change":"2023-09-30T02:30:08.835007+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1250493343}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1978564564}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1806980743}]}]} 2023-09-30T02:30:16.282 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T02:30:16.282 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T02:30:16.282 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:30:16.592 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:16.592 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":16,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:30:15.873202+0000","last_up_change":"2023-09-30T02:30:12.852491+0000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:30:12.888959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:30:14.554460+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115900},{"type":"v1","addr":"172.21.15.115:6801","nonce":115900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115900},{"type":"v1","addr":"172.21.15.115:6803","nonce":115900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115900},{"type":"v1","addr":"172.21.15.115:6807","nonce":115900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115900},{"type":"v1","addr":"172.21.15.115:6805","nonce":115900}]},"public_addr":"172.21.15.115:6801/115900","cluster_addr":"172.21.15.115:6803/115900","heartbeat_back_addr":"172.21.15.115:6807/115900","heartbeat_front_addr":"172.21.15.115:6805/115900","state":["exists","up"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":115902},{"type":"v1","addr":"172.21.15.115:6817","nonce":115902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":115902},{"type":"v1","addr":"172.21.15.115:6820","nonce":115902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":115902},{"type":"v1","addr":"172.21.15.115:6828","nonce":115902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":115902},{"type":"v1","addr":"172.21.15.115:6823","nonce":115902}]},"public_addr":"172.21.15.115:6817/115902","cluster_addr":"172.21.15.115:6820/115902","heartbeat_back_addr":"172.21.15.115:6828/115902","heartbeat_front_addr":"172.21.15.115:6823/115902","state":["exists","up"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6819","nonce":115903},{"type":"v1","addr":"172.21.15.115:6821","nonce":115903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":115903},{"type":"v1","addr":"172.21.15.115:6825","nonce":115903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":115903},{"type":"v1","addr":"172.21.15.115:6831","nonce":115903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6827","nonce":115903},{"type":"v1","addr":"172.21.15.115:6829","nonce":115903}]},"public_addr":"172.21.15.115:6821/115903","cluster_addr":"172.21.15.115:6825/115903","heartbeat_back_addr":"172.21.15.115:6831/115903","heartbeat_front_addr":"172.21.15.115:6829/115903","state":["exists","up"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115901},{"type":"v1","addr":"172.21.15.115:6809","nonce":115901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115901},{"type":"v1","addr":"172.21.15.115:6811","nonce":115901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115901},{"type":"v1","addr":"172.21.15.115:6815","nonce":115901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115901},{"type":"v1","addr":"172.21.15.115:6813","nonce":115901}]},"public_addr":"172.21.15.115:6809/115901","cluster_addr":"172.21.15.115:6811/115901","heartbeat_back_addr":"172.21.15.115:6815/115901","heartbeat_front_addr":"172.21.15.115:6813/115901","state":["exists","up"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115856},{"type":"v1","addr":"172.21.15.157:6809","nonce":115856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115856},{"type":"v1","addr":"172.21.15.157:6811","nonce":115856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115856},{"type":"v1","addr":"172.21.15.157:6815","nonce":115856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115856},{"type":"v1","addr":"172.21.15.157:6813","nonce":115856}]},"public_addr":"172.21.15.157:6809/115856","cluster_addr":"172.21.15.157:6811/115856","heartbeat_back_addr":"172.21.15.157:6815/115856","heartbeat_front_addr":"172.21.15.157:6813/115856","state":["exists","up"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":115886},{"type":"v1","addr":"172.21.15.157:6817","nonce":115886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":115886},{"type":"v1","addr":"172.21.15.157:6819","nonce":115886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":115886},{"type":"v1","addr":"172.21.15.157:6823","nonce":115886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":115886},{"type":"v1","addr":"172.21.15.157:6821","nonce":115886}]},"public_addr":"172.21.15.157:6817/115886","cluster_addr":"172.21.15.157:6819/115886","heartbeat_back_addr":"172.21.15.157:6823/115886","heartbeat_front_addr":"172.21.15.157:6821/115886","state":["exists","up"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":115887},{"type":"v1","addr":"172.21.15.157:6825","nonce":115887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":115887},{"type":"v1","addr":"172.21.15.157:6827","nonce":115887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":115887},{"type":"v1","addr":"172.21.15.157:6831","nonce":115887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":115887},{"type":"v1","addr":"172.21.15.157:6829","nonce":115887}]},"public_addr":"172.21.15.157:6825/115887","cluster_addr":"172.21.15.157:6827/115887","heartbeat_back_addr":"172.21.15.157:6831/115887","heartbeat_front_addr":"172.21.15.157:6829/115887","state":["exists","up"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115857},{"type":"v1","addr":"172.21.15.157:6801","nonce":115857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115857},{"type":"v1","addr":"172.21.15.157:6803","nonce":115857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115857},{"type":"v1","addr":"172.21.15.157:6807","nonce":115857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115857},{"type":"v1","addr":"172.21.15.157:6805","nonce":115857}]},"public_addr":"172.21.15.157:6801/115857","cluster_addr":"172.21.15.157:6803/115857","heartbeat_back_addr":"172.21.15.157:6807/115857","heartbeat_front_addr":"172.21.15.157:6805/115857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.966788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.953004+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.996123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.023275+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.014226+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.941302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.947814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.006181+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:30:16.606 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T02:30:16.606 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:30:16.921 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:16.922 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:30:16.879327+0000","last_up_change":"2023-09-30T02:30:12.852491+0000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:30:12.888959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:30:14.554460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115900},{"type":"v1","addr":"172.21.15.115:6801","nonce":115900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115900},{"type":"v1","addr":"172.21.15.115:6803","nonce":115900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115900},{"type":"v1","addr":"172.21.15.115:6807","nonce":115900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115900},{"type":"v1","addr":"172.21.15.115:6805","nonce":115900}]},"public_addr":"172.21.15.115:6801/115900","cluster_addr":"172.21.15.115:6803/115900","heartbeat_back_addr":"172.21.15.115:6807/115900","heartbeat_front_addr":"172.21.15.115:6805/115900","state":["exists","up"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":115902},{"type":"v1","addr":"172.21.15.115:6817","nonce":115902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":115902},{"type":"v1","addr":"172.21.15.115:6820","nonce":115902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":115902},{"type":"v1","addr":"172.21.15.115:6828","nonce":115902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":115902},{"type":"v1","addr":"172.21.15.115:6823","nonce":115902}]},"public_addr":"172.21.15.115:6817/115902","cluster_addr":"172.21.15.115:6820/115902","heartbeat_back_addr":"172.21.15.115:6828/115902","heartbeat_front_addr":"172.21.15.115:6823/115902","state":["exists","up"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6819","nonce":115903},{"type":"v1","addr":"172.21.15.115:6821","nonce":115903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":115903},{"type":"v1","addr":"172.21.15.115:6825","nonce":115903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":115903},{"type":"v1","addr":"172.21.15.115:6831","nonce":115903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6827","nonce":115903},{"type":"v1","addr":"172.21.15.115:6829","nonce":115903}]},"public_addr":"172.21.15.115:6821/115903","cluster_addr":"172.21.15.115:6825/115903","heartbeat_back_addr":"172.21.15.115:6831/115903","heartbeat_front_addr":"172.21.15.115:6829/115903","state":["exists","up"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115901},{"type":"v1","addr":"172.21.15.115:6809","nonce":115901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115901},{"type":"v1","addr":"172.21.15.115:6811","nonce":115901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115901},{"type":"v1","addr":"172.21.15.115:6815","nonce":115901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115901},{"type":"v1","addr":"172.21.15.115:6813","nonce":115901}]},"public_addr":"172.21.15.115:6809/115901","cluster_addr":"172.21.15.115:6811/115901","heartbeat_back_addr":"172.21.15.115:6815/115901","heartbeat_front_addr":"172.21.15.115:6813/115901","state":["exists","up"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115856},{"type":"v1","addr":"172.21.15.157:6809","nonce":115856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115856},{"type":"v1","addr":"172.21.15.157:6811","nonce":115856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115856},{"type":"v1","addr":"172.21.15.157:6815","nonce":115856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115856},{"type":"v1","addr":"172.21.15.157:6813","nonce":115856}]},"public_addr":"172.21.15.157:6809/115856","cluster_addr":"172.21.15.157:6811/115856","heartbeat_back_addr":"172.21.15.157:6815/115856","heartbeat_front_addr":"172.21.15.157:6813/115856","state":["exists","up"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":115886},{"type":"v1","addr":"172.21.15.157:6817","nonce":115886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":115886},{"type":"v1","addr":"172.21.15.157:6819","nonce":115886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":115886},{"type":"v1","addr":"172.21.15.157:6823","nonce":115886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":115886},{"type":"v1","addr":"172.21.15.157:6821","nonce":115886}]},"public_addr":"172.21.15.157:6817/115886","cluster_addr":"172.21.15.157:6819/115886","heartbeat_back_addr":"172.21.15.157:6823/115886","heartbeat_front_addr":"172.21.15.157:6821/115886","state":["exists","up"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":115887},{"type":"v1","addr":"172.21.15.157:6825","nonce":115887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":115887},{"type":"v1","addr":"172.21.15.157:6827","nonce":115887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":115887},{"type":"v1","addr":"172.21.15.157:6831","nonce":115887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":115887},{"type":"v1","addr":"172.21.15.157:6829","nonce":115887}]},"public_addr":"172.21.15.157:6825/115887","cluster_addr":"172.21.15.157:6827/115887","heartbeat_back_addr":"172.21.15.157:6831/115887","heartbeat_front_addr":"172.21.15.157:6829/115887","state":["exists","up"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115857},{"type":"v1","addr":"172.21.15.157:6801","nonce":115857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115857},{"type":"v1","addr":"172.21.15.157:6803","nonce":115857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115857},{"type":"v1","addr":"172.21.15.157:6807","nonce":115857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115857},{"type":"v1","addr":"172.21.15.157:6805","nonce":115857}]},"public_addr":"172.21.15.157:6801/115857","cluster_addr":"172.21.15.157:6803/115857","heartbeat_back_addr":"172.21.15.157:6807/115857","heartbeat_front_addr":"172.21.15.157:6805/115857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.966788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.953004+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.996123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.023275+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.014226+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.941302+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:10.947814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:30:11.006181+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:30:16.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T02:30:16.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T02:30:16.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T02:30:16.937 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T02:30:16.937 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T02:30:16.937 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T02:30:16.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T02:30:16.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T02:30:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T02:30:17.228 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.228 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T02:30:17.229 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.230 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T02:30:17.231 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T02:30:17.232 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T02:30:17.234 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T02:30:17.248 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.248 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T02:30:17.252 INFO:teuthology.orchestra.run.smithi115.stdout:55834574850 2023-09-30T02:30:17.252 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T02:30:17.839 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.841 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.842 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.857 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.7 2023-09-30T02:30:17.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.6 2023-09-30T02:30:17.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.1 2023-09-30T02:30:17.872 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.877 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.4 2023-09-30T02:30:17.879 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.5 2023-09-30T02:30:17.897 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.0 2023-09-30T02:30:17.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.2 2023-09-30T02:30:17.911 INFO:teuthology.orchestra.run.smithi115.stdout:55834574849 2023-09-30T02:30:17.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.3 2023-09-30T02:30:18.865 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T02:30:18.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T02:30:18.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T02:30:18.878 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T02:30:18.895 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T02:30:18.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T02:30:18.901 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T02:30:18.925 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T02:30:19.438 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.6 2023-09-30T02:30:19.460 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.466 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.475 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.487 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.7 2023-09-30T02:30:19.487 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.1 2023-09-30T02:30:19.496 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.518 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.519 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.5 2023-09-30T02:30:19.534 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.4 2023-09-30T02:30:19.539 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.543 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.2 2023-09-30T02:30:19.563 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.572 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.0 2023-09-30T02:30:19.587 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.596 INFO:teuthology.orchestra.run.smithi115.stdout:55834574851 2023-09-30T02:30:19.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.3 2023-09-30T02:30:19.610 DEBUG:teuthology.parallel:result is None 2023-09-30T02:30:19.610 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T02:30:19.611 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:30:19.961 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:19.962 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:30:19.973 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T02:30:18.841618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":52,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":41596,"kb_used_data":1532,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":754933124,"statfs":{"total":773094113280,"available":773051518976,"internally_reserved":0,"allocated":1568768,"data_stored":671520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.882370+0000","last_change":"2023-09-30T02:30:15.881647+0000","last_active":"2023-09-30T02:30:15.882370+0000","last_peered":"2023-09-30T02:30:15.882370+0000","last_clean":"2023-09-30T02:30:15.882370+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:30:15.882370+0000","last_undegraded":"2023-09-30T02:30:15.882370+0000","last_fullsized":"2023-09-30T02:30:15.882370+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:32.369589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:16.077554+0000","last_change":"2023-09-30T02:30:16.077058+0000","last_active":"2023-09-30T02:30:16.077554+0000","last_peered":"2023-09-30T02:30:16.077554+0000","last_clean":"2023-09-30T02:30:16.077554+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:30:16.077554+0000","last_undegraded":"2023-09-30T02:30:16.077554+0000","last_fullsized":"2023-09-30T02:30:16.077554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:30.744661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.911307+0000","last_change":"2023-09-30T02:30:15.910799+0000","last_active":"2023-09-30T02:30:15.911307+0000","last_peered":"2023-09-30T02:30:15.911307+0000","last_clean":"2023-09-30T02:30:15.911307+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:30:15.911307+0000","last_undegraded":"2023-09-30T02:30:15.911307+0000","last_fullsized":"2023-09-30T02:30:15.911307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:29.957037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.882428+0000","last_change":"2023-09-30T02:30:15.881522+0000","last_active":"2023-09-30T02:30:15.882428+0000","last_peered":"2023-09-30T02:30:15.882428+0000","last_clean":"2023-09-30T02:30:15.882428+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:30:15.882428+0000","last_undegraded":"2023-09-30T02:30:15.882428+0000","last_fullsized":"2023-09-30T02:30:15.882428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:34.919950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'28","reported_seq":50,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T02:30:17.672444+0000","last_change":"2023-09-30T02:30:14.880017+0000","last_active":"2023-09-30T02:30:17.672444+0000","last_peered":"2023-09-30T02:30:17.672444+0000","last_clean":"2023-09-30T02:30:17.672444+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:30:17.672444+0000","last_undegraded":"2023-09-30T02:30:17.672444+0000","last_fullsized":"2023-09-30T02:30:17.672444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_clean_scrub_stamp":"2023-09-30T02:30:13.863889+0000","objects_scrubbed":0,"log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:20.680874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":52,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:16.081034+0000","last_change":"2023-09-30T02:30:16.080566+0000","last_active":"2023-09-30T02:30:16.081034+0000","last_peered":"2023-09-30T02:30:16.081034+0000","last_clean":"2023-09-30T02:30:16.081034+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:30:16.081034+0000","last_undegraded":"2023-09-30T02:30:16.081034+0000","last_fullsized":"2023-09-30T02:30:16.081034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:28.690286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:16.113685+0000","last_change":"2023-09-30T02:30:16.013507+0000","last_active":"2023-09-30T02:30:16.113685+0000","last_peered":"2023-09-30T02:30:16.113685+0000","last_clean":"2023-09-30T02:30:16.113685+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:30:16.113685+0000","last_undegraded":"2023-09-30T02:30:16.113685+0000","last_fullsized":"2023-09-30T02:30:16.113685+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:42.162375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.981921+0000","last_change":"2023-09-30T02:30:15.981377+0000","last_active":"2023-09-30T02:30:15.981921+0000","last_peered":"2023-09-30T02:30:15.981921+0000","last_clean":"2023-09-30T02:30:15.981921+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:30:15.981921+0000","last_undegraded":"2023-09-30T02:30:15.981921+0000","last_fullsized":"2023-09-30T02:30:15.981921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:22.270829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.978103+0000","last_change":"2023-09-30T02:30:15.977018+0000","last_active":"2023-09-30T02:30:15.978103+0000","last_peered":"2023-09-30T02:30:15.978103+0000","last_clean":"2023-09-30T02:30:15.978103+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:30:15.978103+0000","last_undegraded":"2023-09-30T02:30:15.978103+0000","last_fullsized":"2023-09-30T02:30:15.978103+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:39.801662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":52,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":278552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5496,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366344,"statfs":{"total":96636764160,"available":96631136256,"internally_reserved":0,"allocated":311296,"data_stored":191982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.468}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.265000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5496,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366344,"statfs":{"total":96636764160,"available":96631136256,"internally_reserved":0,"allocated":311296,"data_stored":191982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:30:19.975 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:30:20.285 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:20.288 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:30:20.299 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T02:30:18.841618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":52,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":41596,"kb_used_data":1532,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":754933124,"statfs":{"total":773094113280,"available":773051518976,"internally_reserved":0,"allocated":1568768,"data_stored":671520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.882370+0000","last_change":"2023-09-30T02:30:15.881647+0000","last_active":"2023-09-30T02:30:15.882370+0000","last_peered":"2023-09-30T02:30:15.882370+0000","last_clean":"2023-09-30T02:30:15.882370+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:30:15.882370+0000","last_undegraded":"2023-09-30T02:30:15.882370+0000","last_fullsized":"2023-09-30T02:30:15.882370+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:32.369589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:16.077554+0000","last_change":"2023-09-30T02:30:16.077058+0000","last_active":"2023-09-30T02:30:16.077554+0000","last_peered":"2023-09-30T02:30:16.077554+0000","last_clean":"2023-09-30T02:30:16.077554+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:30:16.077554+0000","last_undegraded":"2023-09-30T02:30:16.077554+0000","last_fullsized":"2023-09-30T02:30:16.077554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:30.744661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.911307+0000","last_change":"2023-09-30T02:30:15.910799+0000","last_active":"2023-09-30T02:30:15.911307+0000","last_peered":"2023-09-30T02:30:15.911307+0000","last_clean":"2023-09-30T02:30:15.911307+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:30:15.911307+0000","last_undegraded":"2023-09-30T02:30:15.911307+0000","last_fullsized":"2023-09-30T02:30:15.911307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:29.957037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.882428+0000","last_change":"2023-09-30T02:30:15.881522+0000","last_active":"2023-09-30T02:30:15.882428+0000","last_peered":"2023-09-30T02:30:15.882428+0000","last_clean":"2023-09-30T02:30:15.882428+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:30:15.882428+0000","last_undegraded":"2023-09-30T02:30:15.882428+0000","last_fullsized":"2023-09-30T02:30:15.882428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:34.919950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'28","reported_seq":50,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T02:30:17.672444+0000","last_change":"2023-09-30T02:30:14.880017+0000","last_active":"2023-09-30T02:30:17.672444+0000","last_peered":"2023-09-30T02:30:17.672444+0000","last_clean":"2023-09-30T02:30:17.672444+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:30:17.672444+0000","last_undegraded":"2023-09-30T02:30:17.672444+0000","last_fullsized":"2023-09-30T02:30:17.672444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_clean_scrub_stamp":"2023-09-30T02:30:13.863889+0000","objects_scrubbed":0,"log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:20.680874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":52,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:16.081034+0000","last_change":"2023-09-30T02:30:16.080566+0000","last_active":"2023-09-30T02:30:16.081034+0000","last_peered":"2023-09-30T02:30:16.081034+0000","last_clean":"2023-09-30T02:30:16.081034+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:30:16.081034+0000","last_undegraded":"2023-09-30T02:30:16.081034+0000","last_fullsized":"2023-09-30T02:30:16.081034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:28.690286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:16.113685+0000","last_change":"2023-09-30T02:30:16.013507+0000","last_active":"2023-09-30T02:30:16.113685+0000","last_peered":"2023-09-30T02:30:16.113685+0000","last_clean":"2023-09-30T02:30:16.113685+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:30:16.113685+0000","last_undegraded":"2023-09-30T02:30:16.113685+0000","last_fullsized":"2023-09-30T02:30:16.113685+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:42.162375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.981921+0000","last_change":"2023-09-30T02:30:15.981377+0000","last_active":"2023-09-30T02:30:15.981921+0000","last_peered":"2023-09-30T02:30:15.981921+0000","last_clean":"2023-09-30T02:30:15.981921+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:30:15.981921+0000","last_undegraded":"2023-09-30T02:30:15.981921+0000","last_fullsized":"2023-09-30T02:30:15.981921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:22.270829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T02:30:15.978103+0000","last_change":"2023-09-30T02:30:15.977018+0000","last_active":"2023-09-30T02:30:15.978103+0000","last_peered":"2023-09-30T02:30:15.978103+0000","last_clean":"2023-09-30T02:30:15.978103+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:30:15.978103+0000","last_undegraded":"2023-09-30T02:30:15.978103+0000","last_fullsized":"2023-09-30T02:30:15.978103+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:30:14.869536+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:31:39.801662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":52,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":278552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5496,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366344,"statfs":{"total":96636764160,"available":96631136256,"internally_reserved":0,"allocated":311296,"data_stored":191982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.468}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.265000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5496,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366344,"statfs":{"total":96636764160,"available":96631136256,"internally_reserved":0,"allocated":311296,"data_stored":191982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5100,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366740,"statfs":{"total":96636764160,"available":96631541760,"internally_reserved":0,"allocated":167936,"data_stored":52706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":155648,"data_stored":46970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:30:20.300 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T02:30:20.300 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T02:30:20.300 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T02:30:20.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:20.649 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:20.650 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:23.663 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:24.029 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:24.029 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:27.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:27.406 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:27.407 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:30.423 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:30.788 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:30.789 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:33.802 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:34.163 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:34.164 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:37.178 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:37.540 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:37.541 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:40.555 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:40.917 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:40.917 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:43.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:44.295 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:44.295 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:47.310 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:47.669 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:47.669 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:50.684 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:51.045 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:51.046 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:54.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:54.423 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:54.423 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:30:57.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:30:57.802 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:30:57.802 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:00.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:01.178 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:01.178 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:04.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:04.553 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:07.567 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:07.929 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:07.929 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:10.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:11.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:11.306 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:14.320 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:14.682 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:14.682 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:17.696 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:18.055 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:18.055 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:21.069 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:21.429 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:21.429 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:24.443 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:24.803 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:24.803 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:27.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:28.181 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:28.181 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:31.196 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:31.559 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:31.560 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:34.501 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:31:34.501+0000 7f1b3fbfc700 -1 osd.0 18 handle_osd_map injecting map crc failure 2023-09-30T02:31:34.574 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:34.936 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:34.936 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:37.951 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:37.977 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:31:37.977+0000 7fa7d07ff700 -1 osd.2 20 handle_osd_map injecting map crc failure 2023-09-30T02:31:38.318 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:38.318 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:40.877 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:31:40.877+0000 7f1b3fbfc700 -1 osd.0 21 handle_osd_map injecting map crc failure 2023-09-30T02:31:41.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:41.690 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:41.690 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:44.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:45.068 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:45.068 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:48.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:48.444 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:48.444 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:51.459 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:51.820 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:51.820 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:54.834 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:55.195 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:55.195 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:31:58.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:31:58.578 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:31:58.579 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:01.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:01.955 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:01.955 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:04.968 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:05.330 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:05.330 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:08.344 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:08.705 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:08.705 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:11.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:12.084 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:12.084 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:15.098 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:15.463 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:15.463 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:18.477 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:18.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:18.842 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:21.856 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:22.217 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:22.217 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:25.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:25.594 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:25.594 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:28.608 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:28.971 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:28.971 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:31.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:32.347 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:32.347 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:33.307 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:32:33.307+0000 7ff378058700 -1 osd.5 23 handle_osd_map injecting map crc failure 2023-09-30T02:32:35.361 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:35.452 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T02:32:35.452+0000 7f7e40301700 -1 osd.4 24 handle_osd_map injecting map crc failure 2023-09-30T02:32:35.726 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:35.726 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:38.740 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:39.102 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:39.103 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:40.463 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:32:40.463+0000 7f3ce509b700 -1 osd.6 25 handle_osd_map injecting map crc failure 2023-09-30T02:32:42.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:42.479 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:42.479 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:45.493 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:45.856 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:45.856 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:48.871 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:49.233 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:49.233 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:51.819 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:32:51.819+0000 7ff378058700 -1 osd.5 26 handle_osd_map injecting map crc failure 2023-09-30T02:32:52.248 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:52.612 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:52.613 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:55.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:55.989 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:55.989 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:32:59.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:32:59.367 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:32:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:02.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:02.744 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:02.744 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:03.421 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:33:03.421+0000 7f3510fb3700 -1 osd.1 27 handle_osd_map injecting map crc failure 2023-09-30T02:33:03.422 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:33:03.422+0000 7f3510fb3700 -1 osd.1 27 handle_osd_map injecting map crc failure 2023-09-30T02:33:03.693 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T02:33:03.694+0000 7f5eaf0cf700 -1 osd.3 27 handle_osd_map injecting map crc failure 2023-09-30T02:33:05.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:06.122 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:06.122 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:09.137 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:09.498 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:09.499 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:10.695 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-09-30T02:33:10.695+0000 7f1b3fbfc700 -1 osd.0 29 handle_osd_map injecting map crc failure 2023-09-30T02:33:12.513 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:12.872 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:12.872 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:15.887 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:16.249 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:16.249 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:19.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:19.626 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:19.626 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:22.641 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:23.004 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:23.004 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:26.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:26.381 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:26.381 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:29.395 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:29.756 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:29.756 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:32.770 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:33.135 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:33.135 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:36.150 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:36.516 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:36.517 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:39.018 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:33:39.018+0000 7f3ce3097700 -1 osd.6 30 handle_osd_map injecting map crc failure 2023-09-30T02:33:39.018 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-09-30T02:33:39.018+0000 7fa7d07ff700 -1 osd.2 30 handle_osd_map injecting map crc failure 2023-09-30T02:33:39.530 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:39.891 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:39.891 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:42.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:43.273 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:43.273 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:46.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:46.649 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:46.649 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:49.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:50.024 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:50.024 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:53.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:53.400 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:53.400 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:56.414 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:33:56.774 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:33:56.774 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:33:59.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:00.150 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:00.150 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:03.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:03.523 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:03.523 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:06.538 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:06.603 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T02:34:06.602+0000 7ff378058700 -1 osd.5 35 handle_osd_map injecting map crc failure 2023-09-30T02:34:06.899 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:06.899 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:09.914 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:10.278 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:10.278 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:13.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:13.653 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:13.654 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:16.668 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:17.031 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:17.031 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:20.046 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:20.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:20.410 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:23.424 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:23.789 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:23.789 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:26.804 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:27.167 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:27.167 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:30.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:30.548 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:33.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:33.922 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:33.923 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:36.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:37.301 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:37.301 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:40.315 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:40.678 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:40.679 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:43.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:44.058 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:44.059 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:47.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:47.435 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:47.435 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:50.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:50.813 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:50.813 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:53.828 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:54.193 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:34:57.207 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:34:57.570 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:34:57.570 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:00.584 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:00.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:00.944 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:03.958 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:04.321 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:04.321 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:07.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:07.699 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:07.699 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:09.416 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T02:35:09.416+0000 7f3ce509b700 -1 osd.6 40 handle_osd_map injecting map crc failure 2023-09-30T02:35:09.836 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-09-30T02:35:09.836+0000 7f3510fb3700 -1 osd.1 40 handle_osd_map injecting map crc failure 2023-09-30T02:35:10.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:11.081 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:11.082 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:14.094 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:14.457 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:14.457 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:14.617 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-09-30T02:35:14.617+0000 7f5eaf0cf700 -1 osd.3 42 handle_osd_map injecting map crc failure 2023-09-30T02:35:15.117 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T02:35:15.118+0000 7f5e71c9a700 -1 osd.7 42 handle_osd_map injecting map crc failure 2023-09-30T02:35:17.471 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:17.832 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:17.833 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:20.846 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:35:21.209 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:21.209 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:35:21.223 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T02:35:21.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:35:21.537 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:21.538 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":43,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:35:14.425361+0000","last_up_change":"2023-09-30T02:30:12.852491+0000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:30:12.888959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:30:14.554460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115900},{"type":"v1","addr":"172.21.15.115:6801","nonce":115900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115900},{"type":"v1","addr":"172.21.15.115:6803","nonce":115900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115900},{"type":"v1","addr":"172.21.15.115:6807","nonce":115900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115900},{"type":"v1","addr":"172.21.15.115:6805","nonce":115900}]},"public_addr":"172.21.15.115:6801/115900","cluster_addr":"172.21.15.115:6803/115900","heartbeat_back_addr":"172.21.15.115:6807/115900","heartbeat_front_addr":"172.21.15.115:6805/115900","state":["exists","up"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":115902},{"type":"v1","addr":"172.21.15.115:6817","nonce":115902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":115902},{"type":"v1","addr":"172.21.15.115:6820","nonce":115902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":115902},{"type":"v1","addr":"172.21.15.115:6828","nonce":115902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":115902},{"type":"v1","addr":"172.21.15.115:6823","nonce":115902}]},"public_addr":"172.21.15.115:6817/115902","cluster_addr":"172.21.15.115:6820/115902","heartbeat_back_addr":"172.21.15.115:6828/115902","heartbeat_front_addr":"172.21.15.115:6823/115902","state":["exists","up"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6819","nonce":115903},{"type":"v1","addr":"172.21.15.115:6821","nonce":115903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":115903},{"type":"v1","addr":"172.21.15.115:6825","nonce":115903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":115903},{"type":"v1","addr":"172.21.15.115:6831","nonce":115903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6827","nonce":115903},{"type":"v1","addr":"172.21.15.115:6829","nonce":115903}]},"public_addr":"172.21.15.115:6821/115903","cluster_addr":"172.21.15.115:6825/115903","heartbeat_back_addr":"172.21.15.115:6831/115903","heartbeat_front_addr":"172.21.15.115:6829/115903","state":["exists","up"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115901},{"type":"v1","addr":"172.21.15.115:6809","nonce":115901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115901},{"type":"v1","addr":"172.21.15.115:6811","nonce":115901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115901},{"type":"v1","addr":"172.21.15.115:6815","nonce":115901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115901},{"type":"v1","addr":"172.21.15.115:6813","nonce":115901}]},"public_addr":"172.21.15.115:6809/115901","cluster_addr":"172.21.15.115:6811/115901","heartbeat_back_addr":"172.21.15.115:6815/115901","heartbeat_front_addr":"172.21.15.115:6813/115901","state":["exists","up"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115856},{"type":"v1","addr":"172.21.15.157:6809","nonce":115856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115856},{"type":"v1","addr":"172.21.15.157:6811","nonce":115856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115856},{"type":"v1","addr":"172.21.15.157:6815","nonce":115856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115856},{"type":"v1","addr":"172.21.15.157:6813","nonce":115856}]},"public_addr":"172.21.15.157:6809/115856","cluster_addr":"172.21.15.157:6811/115856","heartbeat_back_addr":"172.21.15.157:6815/115856","heartbeat_front_addr":"172.21.15.157:6813/115856","state":["exists","up"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":115886},{"type":"v1","addr":"172.21.15.157:6817","nonce":115886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":115886},{"type":"v1","addr":"172.21.15.157:6819","nonce":115886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":115886},{"type":"v1","addr":"172.21.15.157:6823","nonce":115886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":115886},{"type":"v1","addr":"172.21.15.157:6821","nonce":115886}]},"public_addr":"172.21.15.157:6817/115886","cluster_addr":"172.21.15.157:6819/115886","heartbeat_back_addr":"172.21.15.157:6823/115886","heartbeat_front_addr":"172.21.15.157:6821/115886","state":["exists","up"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":115887},{"type":"v1","addr":"172.21.15.157:6825","nonce":115887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":115887},{"type":"v1","addr":"172.21.15.157:6827","nonce":115887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":115887},{"type":"v1","addr":"172.21.15.157:6831","nonce":115887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":115887},{"type":"v1","addr":"172.21.15.157:6829","nonce":115887}]},"public_addr":"172.21.15.157:6825/115887","cluster_addr":"172.21.15.157:6827/115887","heartbeat_back_addr":"172.21.15.157:6831/115887","heartbeat_front_addr":"172.21.15.157:6829/115887","state":["exists","up"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115857},{"type":"v1","addr":"172.21.15.157:6801","nonce":115857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115857},{"type":"v1","addr":"172.21.15.157:6803","nonce":115857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115857},{"type":"v1","addr":"172.21.15.157:6807","nonce":115857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115857},{"type":"v1","addr":"172.21.15.157:6805","nonce":115857}]},"public_addr":"172.21.15.157:6801/115857","cluster_addr":"172.21.15.157:6803/115857","heartbeat_back_addr":"172.21.15.157:6807/115857","heartbeat_front_addr":"172.21.15.157:6805/115857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:05.663174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:03.039433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:33:58.914018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:34:28.766204+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:34:16.483756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:10.540245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:13.728841+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:08.575683+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:35:21.551 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T02:35:21.551 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:35:21.858 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:21.860 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:35:21.870 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2023-09-30T02:35:20.902953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":47380,"kb_used_data":4756,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":754927340,"statfs":{"total":773094113280,"available":773045596160,"internally_reserved":0,"allocated":4870144,"data_stored":2607260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.480010"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424741+0000","last_change":"2023-09-30T02:35:07.048881+0000","last_active":"2023-09-30T02:35:11.424741+0000","last_peered":"2023-09-30T02:35:11.424741+0000","last_clean":"2023-09-30T02:35:11.424741+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:35:11.424741+0000","last_undegraded":"2023-09-30T02:35:11.424741+0000","last_fullsized":"2023-09-30T02:35:11.424741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_clean_scrub_stamp":"2023-09-30T02:35:07.048791+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:30.482046+0000","scrub_duration":0.30349070500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424312+0000","last_change":"2023-09-30T02:34:14.080539+0000","last_active":"2023-09-30T02:35:11.424312+0000","last_peered":"2023-09-30T02:35:11.424312+0000","last_clean":"2023-09-30T02:35:11.424312+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:35:11.424312+0000","last_undegraded":"2023-09-30T02:35:11.424312+0000","last_fullsized":"2023-09-30T02:35:11.424312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.080455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.080455+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:41.316039+0000","scrub_duration":0.50328109300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053538+0000","last_change":"2023-09-30T02:34:14.489887+0000","last_active":"2023-09-30T02:35:13.053538+0000","last_peered":"2023-09-30T02:35:13.053538+0000","last_clean":"2023-09-30T02:35:13.053538+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:35:13.053538+0000","last_undegraded":"2023-09-30T02:35:13.053538+0000","last_fullsized":"2023-09-30T02:35:13.053538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.489712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:32:45.821578+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.489712+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:39.582994+0000","scrub_duration":0.502725164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053449+0000","last_change":"2023-09-30T02:34:04.285693+0000","last_active":"2023-09-30T02:35:13.053449+0000","last_peered":"2023-09-30T02:35:13.053449+0000","last_clean":"2023-09-30T02:35:13.053449+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:35:13.053449+0000","last_undegraded":"2023-09-30T02:35:13.053449+0000","last_fullsized":"2023-09-30T02:35:13.053449+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:04.285575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:04.285575+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:20.944000+0000","scrub_duration":0.20249213499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":216,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053538+0000","last_change":"2023-09-30T02:34:01.305491+0000","last_active":"2023-09-30T02:35:13.053538+0000","last_peered":"2023-09-30T02:35:13.053538+0000","last_clean":"2023-09-30T02:35:13.053538+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:35:13.053538+0000","last_undegraded":"2023-09-30T02:35:13.053538+0000","last_fullsized":"2023-09-30T02:35:13.053538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2023-09-30T02:34:01.305372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_clean_scrub_stamp":"2023-09-30T02:34:01.305372+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:27.415083+0000","scrub_duration":0.40451995600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.866877+0000","last_change":"2023-09-30T02:34:27.832803+0000","last_active":"2023-09-30T02:35:12.866877+0000","last_peered":"2023-09-30T02:35:12.866877+0000","last_clean":"2023-09-30T02:35:12.866877+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:35:12.866877+0000","last_undegraded":"2023-09-30T02:35:12.866877+0000","last_fullsized":"2023-09-30T02:35:12.866877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:27.832676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:27.832676+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:35.373737+0000","scrub_duration":0.20230350499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.909951+0000","last_change":"2023-09-30T02:34:20.302744+0000","last_active":"2023-09-30T02:35:11.909951+0000","last_peered":"2023-09-30T02:35:11.909951+0000","last_clean":"2023-09-30T02:35:11.909951+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:35:11.909951+0000","last_undegraded":"2023-09-30T02:35:11.909951+0000","last_fullsized":"2023-09-30T02:35:11.909951+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:20.302648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:20.302648+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:24.481708+0000","scrub_duration":0.30196727200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.666476+0000","last_change":"2023-09-30T02:33:54.052852+0000","last_active":"2023-09-30T02:35:12.666476+0000","last_peered":"2023-09-30T02:35:12.666476+0000","last_clean":"2023-09-30T02:35:12.666476+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:35:12.666476+0000","last_undegraded":"2023-09-30T02:35:12.666476+0000","last_fullsized":"2023-09-30T02:35:12.666476+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:33:54.052650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:33:54.052650+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:21.704144+0000","scrub_duration":0.50227995400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:14.710861+0000","last_change":"2023-09-30T02:35:14.710829+0000","last_active":"2023-09-30T02:35:14.710861+0000","last_peered":"2023-09-30T02:35:14.710861+0000","last_clean":"2023-09-30T02:35:14.710861+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:35:14.710861+0000","last_undegraded":"2023-09-30T02:35:14.710861+0000","last_fullsized":"2023-09-30T02:35:14.710861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_clean_scrub_stamp":"2023-09-30T02:35:14.710754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:41.843718+0000","scrub_duration":0.20152634699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6688,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365152,"statfs":{"total":96636764160,"available":96629915648,"internally_reserved":0,"allocated":1007616,"data_stored":722574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.621,"15min":0.68799999999999994},"min":{"1min":0.41599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.745,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.05,"15min":1.5349999999999999},"min":{"1min":0.44800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":22.445,"15min":22.445},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":1.173,"15min":1.7769999999999999},"min":{"1min":0.46500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":22.483000000000001,"15min":22.483000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79400000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.626,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":3.9870000000000001,"15min":3.9870000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.0169999999999999,"15min":1.4370000000000001},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81999999999999995,"5min":22.547999999999998,"15min":22.547999999999998},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63100000000000001,"15min":0.69499999999999995},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81000000000000005,"5min":3.9079999999999999,"15min":3.9079999999999999},"last":0.623},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.232,"15min":1.859},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.87,"5min":22.515000000000001,"15min":22.515000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68400000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.495,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.63900000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.93200000000000005,"15min":1.367},"min":{"1min":0.35399999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":22.398,"15min":22.398},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54900000000000004,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.72799999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.36399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55400000000000005,"15min":0.51400000000000001},"min":{"1min":0.42899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70199999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.302,"15min":6.96},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71999999999999997,"5min":175.99700000000001,"15min":175.99700000000001},"last":0.70899999999999996}]},{"osd":1,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.48399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66000000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.504,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.747},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.55500000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.753,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71799999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.624}]},{"osd":7,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.2889999999999999,"15min":3.2799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":57.932000000000002,"15min":57.932000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.355,"15min":3.335},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":57.914000000000001,"15min":57.914000000000001},"last":0.73199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5728,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366112,"statfs":{"total":96636764160,"available":96630898688,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.372,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56499999999999995,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.35599999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42899999999999999,"15min":0.42399999999999999},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64400000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.996,"15min":0.996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.505,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.72999999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.748,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.53900000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.621},"min":{"1min":0.49399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81399999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6680,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365160,"statfs":{"total":96636764160,"available":96629923840,"internally_reserved":0,"allocated":999424,"data_stored":721952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.63100000000000001,"5min":1.629,"15min":1.629},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.39400000000000002,"15min":0.42899999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.48199999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41699999999999998,"15min":0.44800000000000001},"min":{"1min":0.255,"5min":0.253,"15min":0.253},"max":{"1min":0.498,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.43099999999999999,"15min":0.46200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57499999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.32200000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":1.107,"15min":1.486},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.56000000000000005,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46000000000000002,"15min":0.495},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59899999999999998,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.747,"5min":1.361,"15min":1.361},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.319,"15min":1.694},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":26.218,"15min":26.218},"last":0.55200000000000005}]},{"osd":5,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.355,"15min":1.73},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":26.247,"15min":26.247},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.44500000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81200000000000006,"5min":1.583,"15min":1.583},"last":0.72999999999999998}]},{"osd":6,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65100000000000002,"15min":0.68300000000000005},"min":{"1min":0.48199999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.748,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.65000000000000002,"15min":0.66900000000000004},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81200000000000006,"5min":1.409,"15min":1.409},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.82599999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.627,"5min":1.3540000000000001,"15min":1.7470000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88700000000000001,"5min":26.271999999999998,"15min":26.271999999999998},"last":0.52600000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":475136,"data_stored":193646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":2.1619999999999999,"15min":3.952},"min":{"1min":0.222,"5min":0.217,"15min":0.217},"max":{"1min":0.55600000000000005,"5min":81.966999999999999,"15min":81.966999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42299999999999999,"15min":0.46300000000000002},"min":{"1min":0.25,"5min":0.218,"15min":0.218},"max":{"1min":0.59499999999999997,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.59299999999999997,"15min":0.80500000000000005},"min":{"1min":0.252,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61499999999999999,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.72299999999999998,"15min":1.0469999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53400000000000003,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.64300000000000002,"15min":0.89500000000000002},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.52800000000000002,"5min":12.164,"15min":12.164},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.621,"15min":0.81100000000000005},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65900000000000003,"5min":7.101,"15min":7.101},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.65900000000000003,"15min":0.70399999999999996},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":2.7639999999999998,"15min":2.7639999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.627,"15min":0.64400000000000002},"min":{"1min":0.51200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93999999999999995,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0169999999999999,"15min":1.401},"min":{"1min":0.48699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":12.07,"15min":12.07},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.877,"15min":1.0920000000000001},"min":{"1min":0.52500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":10.353999999999999,"15min":10.353999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90100000000000002,"5min":1.103,"15min":1.103},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.872,"15min":1.123},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":12.135,"15min":12.135},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.72699999999999998,"15min":0.81299999999999994},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85299999999999998,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.95099999999999996,"15min":1.2789999999999999},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78400000000000003,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.70599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5664,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366176,"statfs":{"total":96636764160,"available":96630964224,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36899999999999999,"15min":0.36399999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.621,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.59099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.27000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.503,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.317},{"interface":"front","average":{"1min":0.378,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.52900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.25}]},{"osd":2,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.32300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47299999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57299999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.37}]},{"osd":4,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76000000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80900000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83799999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79000000000000004,"5min":1.006,"15min":1.006},"last":0.73399999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":1.268,"15min":0.83099999999999996},"min":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":65.600999999999999,"5min":65.600999999999999,"15min":65.600999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.51700000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80100000000000005,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.65700000000000003},"min":{"1min":0.50900000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.755,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5660,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366180,"statfs":{"total":96636764160,"available":96630968320,"internally_reserved":0,"allocated":479232,"data_stored":193932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66800000000000004,"5min":0.754,"15min":0.754},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.38300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.55600000000000005},"min":{"1min":0.437,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.42999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.51100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82899999999999996,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.496,"15min":0.47599999999999998},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.433}]},{"osd":6,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75800000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":4.883,"15min":9.2129999999999992},"min":{"1min":0.377,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73599999999999999,"5min":192.72,"15min":192.72},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.623,"15min":0.68600000000000005},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79800000000000004,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.75600000000000001},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81000000000000005,"5min":4.5,"15min":4.5},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.61299999999999999,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.67700000000000005},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72299999999999998,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66100000000000003,"15min":0.74099999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.73699999999999999,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.496},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.67500000000000004,"15min":0.78100000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.75600000000000001,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":2.774,"15min":2.774},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.59799999999999998},"min":{"1min":0.35599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73899999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.66200000000000003}]},{"osd":6,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58099999999999996,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64700000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60899999999999999,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71399999999999997,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.4059999999999999,"15min":2.2050000000000001},"min":{"1min":0.502,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69199999999999995,"5min":33.798000000000002,"15min":33.798000000000002},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:35:21.873 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:35:22.184 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:22.185 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:35:22.196 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2023-09-30T02:35:20.902953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":47380,"kb_used_data":4756,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":754927340,"statfs":{"total":773094113280,"available":773045596160,"internally_reserved":0,"allocated":4870144,"data_stored":2607260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.480010"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424741+0000","last_change":"2023-09-30T02:35:07.048881+0000","last_active":"2023-09-30T02:35:11.424741+0000","last_peered":"2023-09-30T02:35:11.424741+0000","last_clean":"2023-09-30T02:35:11.424741+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:35:11.424741+0000","last_undegraded":"2023-09-30T02:35:11.424741+0000","last_fullsized":"2023-09-30T02:35:11.424741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_clean_scrub_stamp":"2023-09-30T02:35:07.048791+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:30.482046+0000","scrub_duration":0.30349070500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424312+0000","last_change":"2023-09-30T02:34:14.080539+0000","last_active":"2023-09-30T02:35:11.424312+0000","last_peered":"2023-09-30T02:35:11.424312+0000","last_clean":"2023-09-30T02:35:11.424312+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:35:11.424312+0000","last_undegraded":"2023-09-30T02:35:11.424312+0000","last_fullsized":"2023-09-30T02:35:11.424312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.080455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.080455+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:41.316039+0000","scrub_duration":0.50328109300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053538+0000","last_change":"2023-09-30T02:34:14.489887+0000","last_active":"2023-09-30T02:35:13.053538+0000","last_peered":"2023-09-30T02:35:13.053538+0000","last_clean":"2023-09-30T02:35:13.053538+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:35:13.053538+0000","last_undegraded":"2023-09-30T02:35:13.053538+0000","last_fullsized":"2023-09-30T02:35:13.053538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.489712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:32:45.821578+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.489712+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:39.582994+0000","scrub_duration":0.502725164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053449+0000","last_change":"2023-09-30T02:34:04.285693+0000","last_active":"2023-09-30T02:35:13.053449+0000","last_peered":"2023-09-30T02:35:13.053449+0000","last_clean":"2023-09-30T02:35:13.053449+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:35:13.053449+0000","last_undegraded":"2023-09-30T02:35:13.053449+0000","last_fullsized":"2023-09-30T02:35:13.053449+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:04.285575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:04.285575+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:20.944000+0000","scrub_duration":0.20249213499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":216,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053538+0000","last_change":"2023-09-30T02:34:01.305491+0000","last_active":"2023-09-30T02:35:13.053538+0000","last_peered":"2023-09-30T02:35:13.053538+0000","last_clean":"2023-09-30T02:35:13.053538+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:35:13.053538+0000","last_undegraded":"2023-09-30T02:35:13.053538+0000","last_fullsized":"2023-09-30T02:35:13.053538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2023-09-30T02:34:01.305372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_clean_scrub_stamp":"2023-09-30T02:34:01.305372+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:27.415083+0000","scrub_duration":0.40451995600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.866877+0000","last_change":"2023-09-30T02:34:27.832803+0000","last_active":"2023-09-30T02:35:12.866877+0000","last_peered":"2023-09-30T02:35:12.866877+0000","last_clean":"2023-09-30T02:35:12.866877+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:35:12.866877+0000","last_undegraded":"2023-09-30T02:35:12.866877+0000","last_fullsized":"2023-09-30T02:35:12.866877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:27.832676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:27.832676+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:35.373737+0000","scrub_duration":0.20230350499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.909951+0000","last_change":"2023-09-30T02:34:20.302744+0000","last_active":"2023-09-30T02:35:11.909951+0000","last_peered":"2023-09-30T02:35:11.909951+0000","last_clean":"2023-09-30T02:35:11.909951+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:35:11.909951+0000","last_undegraded":"2023-09-30T02:35:11.909951+0000","last_fullsized":"2023-09-30T02:35:11.909951+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:20.302648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:20.302648+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:24.481708+0000","scrub_duration":0.30196727200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.666476+0000","last_change":"2023-09-30T02:33:54.052852+0000","last_active":"2023-09-30T02:35:12.666476+0000","last_peered":"2023-09-30T02:35:12.666476+0000","last_clean":"2023-09-30T02:35:12.666476+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:35:12.666476+0000","last_undegraded":"2023-09-30T02:35:12.666476+0000","last_fullsized":"2023-09-30T02:35:12.666476+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:33:54.052650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:33:54.052650+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:21.704144+0000","scrub_duration":0.50227995400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:14.710861+0000","last_change":"2023-09-30T02:35:14.710829+0000","last_active":"2023-09-30T02:35:14.710861+0000","last_peered":"2023-09-30T02:35:14.710861+0000","last_clean":"2023-09-30T02:35:14.710861+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:35:14.710861+0000","last_undegraded":"2023-09-30T02:35:14.710861+0000","last_fullsized":"2023-09-30T02:35:14.710861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_clean_scrub_stamp":"2023-09-30T02:35:14.710754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:41.843718+0000","scrub_duration":0.20152634699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6688,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365152,"statfs":{"total":96636764160,"available":96629915648,"internally_reserved":0,"allocated":1007616,"data_stored":722574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.621,"15min":0.68799999999999994},"min":{"1min":0.41599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.745,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.05,"15min":1.5349999999999999},"min":{"1min":0.44800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":22.445,"15min":22.445},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":1.173,"15min":1.7769999999999999},"min":{"1min":0.46500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":22.483000000000001,"15min":22.483000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79400000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.626,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":3.9870000000000001,"15min":3.9870000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.0169999999999999,"15min":1.4370000000000001},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81999999999999995,"5min":22.547999999999998,"15min":22.547999999999998},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63100000000000001,"15min":0.69499999999999995},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81000000000000005,"5min":3.9079999999999999,"15min":3.9079999999999999},"last":0.623},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.232,"15min":1.859},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.87,"5min":22.515000000000001,"15min":22.515000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68400000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.495,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.63900000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.93200000000000005,"15min":1.367},"min":{"1min":0.35399999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":22.398,"15min":22.398},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54900000000000004,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.72799999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.36399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55400000000000005,"15min":0.51400000000000001},"min":{"1min":0.42899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70199999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.302,"15min":6.96},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71999999999999997,"5min":175.99700000000001,"15min":175.99700000000001},"last":0.70899999999999996}]},{"osd":1,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.48399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66000000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.504,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.747},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.55500000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.753,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71799999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.624}]},{"osd":7,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.2889999999999999,"15min":3.2799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":57.932000000000002,"15min":57.932000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.355,"15min":3.335},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":57.914000000000001,"15min":57.914000000000001},"last":0.73199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5728,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366112,"statfs":{"total":96636764160,"available":96630898688,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.372,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56499999999999995,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.35599999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42899999999999999,"15min":0.42399999999999999},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64400000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.996,"15min":0.996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.505,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.72999999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.748,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.53900000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.621},"min":{"1min":0.49399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81399999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6680,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365160,"statfs":{"total":96636764160,"available":96629923840,"internally_reserved":0,"allocated":999424,"data_stored":721952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.63100000000000001,"5min":1.629,"15min":1.629},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.39400000000000002,"15min":0.42899999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.48199999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41699999999999998,"15min":0.44800000000000001},"min":{"1min":0.255,"5min":0.253,"15min":0.253},"max":{"1min":0.498,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.43099999999999999,"15min":0.46200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57499999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.32200000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":1.107,"15min":1.486},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.56000000000000005,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46000000000000002,"15min":0.495},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59899999999999998,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.747,"5min":1.361,"15min":1.361},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.319,"15min":1.694},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":26.218,"15min":26.218},"last":0.55200000000000005}]},{"osd":5,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.355,"15min":1.73},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":26.247,"15min":26.247},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.44500000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81200000000000006,"5min":1.583,"15min":1.583},"last":0.72999999999999998}]},{"osd":6,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65100000000000002,"15min":0.68300000000000005},"min":{"1min":0.48199999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.748,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.65000000000000002,"15min":0.66900000000000004},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81200000000000006,"5min":1.409,"15min":1.409},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.82599999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.627,"5min":1.3540000000000001,"15min":1.7470000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88700000000000001,"5min":26.271999999999998,"15min":26.271999999999998},"last":0.52600000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":475136,"data_stored":193646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":2.1619999999999999,"15min":3.952},"min":{"1min":0.222,"5min":0.217,"15min":0.217},"max":{"1min":0.55600000000000005,"5min":81.966999999999999,"15min":81.966999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42299999999999999,"15min":0.46300000000000002},"min":{"1min":0.25,"5min":0.218,"15min":0.218},"max":{"1min":0.59499999999999997,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.59299999999999997,"15min":0.80500000000000005},"min":{"1min":0.252,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61499999999999999,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.72299999999999998,"15min":1.0469999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53400000000000003,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.64300000000000002,"15min":0.89500000000000002},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.52800000000000002,"5min":12.164,"15min":12.164},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.621,"15min":0.81100000000000005},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65900000000000003,"5min":7.101,"15min":7.101},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.65900000000000003,"15min":0.70399999999999996},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":2.7639999999999998,"15min":2.7639999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.627,"15min":0.64400000000000002},"min":{"1min":0.51200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93999999999999995,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0169999999999999,"15min":1.401},"min":{"1min":0.48699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":12.07,"15min":12.07},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.877,"15min":1.0920000000000001},"min":{"1min":0.52500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":10.353999999999999,"15min":10.353999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90100000000000002,"5min":1.103,"15min":1.103},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.872,"15min":1.123},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":12.135,"15min":12.135},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.72699999999999998,"15min":0.81299999999999994},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85299999999999998,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.95099999999999996,"15min":1.2789999999999999},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78400000000000003,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.70599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5664,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366176,"statfs":{"total":96636764160,"available":96630964224,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36899999999999999,"15min":0.36399999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.621,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.59099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.27000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.503,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.317},{"interface":"front","average":{"1min":0.378,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.52900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.25}]},{"osd":2,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.32300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47299999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57299999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.37}]},{"osd":4,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76000000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80900000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83799999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79000000000000004,"5min":1.006,"15min":1.006},"last":0.73399999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":1.268,"15min":0.83099999999999996},"min":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":65.600999999999999,"5min":65.600999999999999,"15min":65.600999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.51700000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80100000000000005,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.65700000000000003},"min":{"1min":0.50900000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.755,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5660,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366180,"statfs":{"total":96636764160,"available":96630968320,"internally_reserved":0,"allocated":479232,"data_stored":193932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66800000000000004,"5min":0.754,"15min":0.754},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.38300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.55600000000000005},"min":{"1min":0.437,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.42999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.51100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82899999999999996,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.496,"15min":0.47599999999999998},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.433}]},{"osd":6,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75800000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":4.883,"15min":9.2129999999999992},"min":{"1min":0.377,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73599999999999999,"5min":192.72,"15min":192.72},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.623,"15min":0.68600000000000005},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79800000000000004,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.75600000000000001},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81000000000000005,"5min":4.5,"15min":4.5},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.61299999999999999,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.67700000000000005},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72299999999999998,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66100000000000003,"15min":0.74099999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.73699999999999999,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.496},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.67500000000000004,"15min":0.78100000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.75600000000000001,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":2.774,"15min":2.774},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.59799999999999998},"min":{"1min":0.35599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73899999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.66200000000000003}]},{"osd":6,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58099999999999996,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64700000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60899999999999999,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71399999999999997,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.4059999999999999,"15min":2.2050000000000001},"min":{"1min":0.502,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69199999999999995,"5min":33.798000000000002,"15min":33.798000000000002},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:35:22.199 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T02:35:22.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:35:22.457 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T02:35:22.457+0000 7f5e73c9e700 -1 osd.7 43 handle_osd_map injecting map crc failure 2023-09-30T02:35:22.458 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T02:35:22.457+0000 7f7e40301700 -1 osd.4 43 handle_osd_map injecting map crc failure 2023-09-30T02:35:22.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:22.509 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:35:22.519 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2023-09-30T02:35:22.458271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":47380,"kb_used_data":4756,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":754927340,"statfs":{"total":773094113280,"available":773045596160,"internally_reserved":0,"allocated":4870144,"data_stored":2607260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.556794"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424741+0000","last_change":"2023-09-30T02:35:07.048881+0000","last_active":"2023-09-30T02:35:11.424741+0000","last_peered":"2023-09-30T02:35:11.424741+0000","last_clean":"2023-09-30T02:35:11.424741+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:35:11.424741+0000","last_undegraded":"2023-09-30T02:35:11.424741+0000","last_fullsized":"2023-09-30T02:35:11.424741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_clean_scrub_stamp":"2023-09-30T02:35:07.048791+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:30.482046+0000","scrub_duration":0.30349070500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424312+0000","last_change":"2023-09-30T02:34:14.080539+0000","last_active":"2023-09-30T02:35:11.424312+0000","last_peered":"2023-09-30T02:35:11.424312+0000","last_clean":"2023-09-30T02:35:11.424312+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:35:11.424312+0000","last_undegraded":"2023-09-30T02:35:11.424312+0000","last_fullsized":"2023-09-30T02:35:11.424312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.080455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.080455+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:41.316039+0000","scrub_duration":0.50328109300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053538+0000","last_change":"2023-09-30T02:34:14.489887+0000","last_active":"2023-09-30T02:35:13.053538+0000","last_peered":"2023-09-30T02:35:13.053538+0000","last_clean":"2023-09-30T02:35:13.053538+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:35:13.053538+0000","last_undegraded":"2023-09-30T02:35:13.053538+0000","last_fullsized":"2023-09-30T02:35:13.053538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.489712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:32:45.821578+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.489712+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:39.582994+0000","scrub_duration":0.502725164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053449+0000","last_change":"2023-09-30T02:34:04.285693+0000","last_active":"2023-09-30T02:35:13.053449+0000","last_peered":"2023-09-30T02:35:13.053449+0000","last_clean":"2023-09-30T02:35:13.053449+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:35:13.053449+0000","last_undegraded":"2023-09-30T02:35:13.053449+0000","last_fullsized":"2023-09-30T02:35:13.053449+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:04.285575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:04.285575+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:20.944000+0000","scrub_duration":0.20249213499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":216,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:13.053538+0000","last_change":"2023-09-30T02:34:01.305491+0000","last_active":"2023-09-30T02:35:13.053538+0000","last_peered":"2023-09-30T02:35:13.053538+0000","last_clean":"2023-09-30T02:35:13.053538+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:35:13.053538+0000","last_undegraded":"2023-09-30T02:35:13.053538+0000","last_fullsized":"2023-09-30T02:35:13.053538+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2023-09-30T02:34:01.305372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_clean_scrub_stamp":"2023-09-30T02:34:01.305372+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:27.415083+0000","scrub_duration":0.40451995600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.866877+0000","last_change":"2023-09-30T02:34:27.832803+0000","last_active":"2023-09-30T02:35:12.866877+0000","last_peered":"2023-09-30T02:35:12.866877+0000","last_clean":"2023-09-30T02:35:12.866877+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:35:12.866877+0000","last_undegraded":"2023-09-30T02:35:12.866877+0000","last_fullsized":"2023-09-30T02:35:12.866877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:27.832676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:27.832676+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:35.373737+0000","scrub_duration":0.20230350499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.909951+0000","last_change":"2023-09-30T02:34:20.302744+0000","last_active":"2023-09-30T02:35:11.909951+0000","last_peered":"2023-09-30T02:35:11.909951+0000","last_clean":"2023-09-30T02:35:11.909951+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:35:11.909951+0000","last_undegraded":"2023-09-30T02:35:11.909951+0000","last_fullsized":"2023-09-30T02:35:11.909951+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:20.302648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:20.302648+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:24.481708+0000","scrub_duration":0.30196727200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.666476+0000","last_change":"2023-09-30T02:33:54.052852+0000","last_active":"2023-09-30T02:35:12.666476+0000","last_peered":"2023-09-30T02:35:12.666476+0000","last_clean":"2023-09-30T02:35:12.666476+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:35:12.666476+0000","last_undegraded":"2023-09-30T02:35:12.666476+0000","last_fullsized":"2023-09-30T02:35:12.666476+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:33:54.052650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:33:54.052650+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:21.704144+0000","scrub_duration":0.50227995400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:14.710861+0000","last_change":"2023-09-30T02:35:14.710829+0000","last_active":"2023-09-30T02:35:14.710861+0000","last_peered":"2023-09-30T02:35:14.710861+0000","last_clean":"2023-09-30T02:35:14.710861+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:35:14.710861+0000","last_undegraded":"2023-09-30T02:35:14.710861+0000","last_fullsized":"2023-09-30T02:35:14.710861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_clean_scrub_stamp":"2023-09-30T02:35:14.710754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:41.843718+0000","scrub_duration":0.20152634699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6688,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365152,"statfs":{"total":96636764160,"available":96629915648,"internally_reserved":0,"allocated":1007616,"data_stored":722574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.621,"15min":0.68799999999999994},"min":{"1min":0.41599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.745,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.05,"15min":1.5349999999999999},"min":{"1min":0.44800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":22.445,"15min":22.445},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":1.173,"15min":1.7769999999999999},"min":{"1min":0.46500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":22.483000000000001,"15min":22.483000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39800000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79400000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.626,"15min":0.68600000000000005},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":3.9870000000000001,"15min":3.9870000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.0169999999999999,"15min":1.4370000000000001},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81999999999999995,"5min":22.547999999999998,"15min":22.547999999999998},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63100000000000001,"15min":0.69499999999999995},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81000000000000005,"5min":3.9079999999999999,"15min":3.9079999999999999},"last":0.623},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.232,"15min":1.859},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.87,"5min":22.515000000000001,"15min":22.515000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68400000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.495,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.63900000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.93200000000000005,"15min":1.367},"min":{"1min":0.35399999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":22.398,"15min":22.398},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58199999999999996,"15min":0.58999999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54900000000000004,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.72799999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.36399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55400000000000005,"15min":0.51400000000000001},"min":{"1min":0.42899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70199999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.302,"15min":6.96},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71999999999999997,"5min":175.99700000000001,"15min":175.99700000000001},"last":0.70899999999999996}]},{"osd":1,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.48399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66000000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.504,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.747},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.55500000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.753,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71799999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.624}]},{"osd":7,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.2889999999999999,"15min":3.2799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":57.932000000000002,"15min":57.932000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.355,"15min":3.335},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":57.914000000000001,"15min":57.914000000000001},"last":0.73199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5728,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366112,"statfs":{"total":96636764160,"available":96630898688,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.372,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56499999999999995,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.35599999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42899999999999999,"15min":0.42399999999999999},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64400000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.996,"15min":0.996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.505,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.72999999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.748,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.53900000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.621},"min":{"1min":0.49399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81399999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6680,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365160,"statfs":{"total":96636764160,"available":96629923840,"internally_reserved":0,"allocated":999424,"data_stored":721952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.63100000000000001,"5min":1.629,"15min":1.629},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.39400000000000002,"15min":0.42899999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.48199999999999998,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41699999999999998,"15min":0.44800000000000001},"min":{"1min":0.255,"5min":0.253,"15min":0.253},"max":{"1min":0.498,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.43099999999999999,"15min":0.46200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57499999999999996,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.32200000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":1.107,"15min":1.486},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.56000000000000005,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46000000000000002,"15min":0.495},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59899999999999998,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61199999999999999,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.747,"5min":1.361,"15min":1.361},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.319,"15min":1.694},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":26.218,"15min":26.218},"last":0.55200000000000005}]},{"osd":5,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.355,"15min":1.73},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":26.247,"15min":26.247},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.44500000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81200000000000006,"5min":1.583,"15min":1.583},"last":0.72999999999999998}]},{"osd":6,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65100000000000002,"15min":0.68300000000000005},"min":{"1min":0.48199999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.748,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.65000000000000002,"15min":0.66900000000000004},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81200000000000006,"5min":1.409,"15min":1.409},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.67400000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.82599999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.627,"5min":1.3540000000000001,"15min":1.7470000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88700000000000001,"5min":26.271999999999998,"15min":26.271999999999998},"last":0.52600000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":475136,"data_stored":193646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":2.1619999999999999,"15min":3.952},"min":{"1min":0.222,"5min":0.217,"15min":0.217},"max":{"1min":0.55600000000000005,"5min":81.966999999999999,"15min":81.966999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42299999999999999,"15min":0.46300000000000002},"min":{"1min":0.25,"5min":0.218,"15min":0.218},"max":{"1min":0.59499999999999997,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.59299999999999997,"15min":0.80500000000000005},"min":{"1min":0.252,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61499999999999999,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.72299999999999998,"15min":1.0469999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53400000000000003,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.64300000000000002,"15min":0.89500000000000002},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.52800000000000002,"5min":12.164,"15min":12.164},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.621,"15min":0.81100000000000005},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65900000000000003,"5min":7.101,"15min":7.101},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.65900000000000003,"15min":0.70399999999999996},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":2.7639999999999998,"15min":2.7639999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.627,"15min":0.64400000000000002},"min":{"1min":0.51200000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93999999999999995,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0169999999999999,"15min":1.401},"min":{"1min":0.48699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":12.07,"15min":12.07},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.877,"15min":1.0920000000000001},"min":{"1min":0.52500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":10.353999999999999,"15min":10.353999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.44700000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90100000000000002,"5min":1.103,"15min":1.103},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.872,"15min":1.123},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73899999999999999,"5min":12.135,"15min":12.135},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.72699999999999998,"15min":0.81299999999999994},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85299999999999998,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.95099999999999996,"15min":1.2789999999999999},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78400000000000003,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.70599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5664,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366176,"statfs":{"total":96636764160,"available":96630964224,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36899999999999999,"15min":0.36399999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.621,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.59099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.27000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.503,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.317},{"interface":"front","average":{"1min":0.378,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.52900000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.25}]},{"osd":2,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.32300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47299999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57299999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.37}]},{"osd":4,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.46100000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76000000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80900000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83799999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79000000000000004,"5min":1.006,"15min":1.006},"last":0.73399999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":1.268,"15min":0.83099999999999996},"min":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":65.600999999999999,"5min":65.600999999999999,"15min":65.600999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65500000000000003,"15min":0.67000000000000004},"min":{"1min":0.51700000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80100000000000005,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Sep 30 02:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.65700000000000003},"min":{"1min":0.50900000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81499999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.65100000000000002},"min":{"1min":0.48799999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.755,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5660,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366180,"statfs":{"total":96636764160,"available":96630968320,"internally_reserved":0,"allocated":479232,"data_stored":193932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66800000000000004,"5min":0.754,"15min":0.754},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.38300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.55600000000000005},"min":{"1min":0.437,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.42999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.51100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82899999999999996,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.496,"15min":0.47599999999999998},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.433}]},{"osd":6,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75800000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":4.883,"15min":9.2129999999999992},"min":{"1min":0.377,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73599999999999999,"5min":192.72,"15min":192.72},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.623,"15min":0.68600000000000005},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79800000000000004,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.75600000000000001},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81000000000000005,"5min":4.5,"15min":4.5},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.61299999999999999,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.67700000000000005},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72299999999999998,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66100000000000003,"15min":0.74099999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.73699999999999999,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.496},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.67500000000000004,"15min":0.78100000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.75600000000000001,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":2.774,"15min":2.774},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.59799999999999998},"min":{"1min":0.35599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73899999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.66200000000000003}]},{"osd":6,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58099999999999996,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64700000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60899999999999999,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71399999999999997,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.4059999999999999,"15min":2.2050000000000001},"min":{"1min":0.502,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69199999999999995,"5min":33.798000000000002,"15min":33.798000000000002},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:35:22.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:35:22.833 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:22.833 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":44,"fsid":"e8e31025-f333-40f5-9e8c-07777b1949b2","created":"2023-09-30T02:30:03.404129+0000","modified":"2023-09-30T02:35:22.451728+0000","last_up_change":"2023-09-30T02:30:12.852491+0000","last_in_change":"2023-09-30T02:30:05.813739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:30:12.888959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:30:14.554460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a0eead8b-1085-45e3-82b6-6fdbbeda801a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":115900},{"type":"v1","addr":"172.21.15.115:6801","nonce":115900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":115900},{"type":"v1","addr":"172.21.15.115:6803","nonce":115900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":115900},{"type":"v1","addr":"172.21.15.115:6807","nonce":115900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":115900},{"type":"v1","addr":"172.21.15.115:6805","nonce":115900}]},"public_addr":"172.21.15.115:6801/115900","cluster_addr":"172.21.15.115:6803/115900","heartbeat_back_addr":"172.21.15.115:6807/115900","heartbeat_front_addr":"172.21.15.115:6805/115900","state":["exists","up"]},{"osd":1,"uuid":"a459ef89-74f1-472d-a093-6020937471e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":115902},{"type":"v1","addr":"172.21.15.115:6817","nonce":115902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":115902},{"type":"v1","addr":"172.21.15.115:6820","nonce":115902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":115902},{"type":"v1","addr":"172.21.15.115:6828","nonce":115902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":115902},{"type":"v1","addr":"172.21.15.115:6823","nonce":115902}]},"public_addr":"172.21.15.115:6817/115902","cluster_addr":"172.21.15.115:6820/115902","heartbeat_back_addr":"172.21.15.115:6828/115902","heartbeat_front_addr":"172.21.15.115:6823/115902","state":["exists","up"]},{"osd":2,"uuid":"d84ba82d-49e1-4cc4-bdfa-eb71a4f118d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6819","nonce":115903},{"type":"v1","addr":"172.21.15.115:6821","nonce":115903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":115903},{"type":"v1","addr":"172.21.15.115:6825","nonce":115903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":115903},{"type":"v1","addr":"172.21.15.115:6831","nonce":115903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6827","nonce":115903},{"type":"v1","addr":"172.21.15.115:6829","nonce":115903}]},"public_addr":"172.21.15.115:6821/115903","cluster_addr":"172.21.15.115:6825/115903","heartbeat_back_addr":"172.21.15.115:6831/115903","heartbeat_front_addr":"172.21.15.115:6829/115903","state":["exists","up"]},{"osd":3,"uuid":"85677263-5295-4c4b-bb83-1b01ba58356c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":115901},{"type":"v1","addr":"172.21.15.115:6809","nonce":115901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":115901},{"type":"v1","addr":"172.21.15.115:6811","nonce":115901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":115901},{"type":"v1","addr":"172.21.15.115:6815","nonce":115901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":115901},{"type":"v1","addr":"172.21.15.115:6813","nonce":115901}]},"public_addr":"172.21.15.115:6809/115901","cluster_addr":"172.21.15.115:6811/115901","heartbeat_back_addr":"172.21.15.115:6815/115901","heartbeat_front_addr":"172.21.15.115:6813/115901","state":["exists","up"]},{"osd":4,"uuid":"2cce4678-d9e6-411e-8fe1-3b8aef44ab18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":115856},{"type":"v1","addr":"172.21.15.157:6809","nonce":115856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":115856},{"type":"v1","addr":"172.21.15.157:6811","nonce":115856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":115856},{"type":"v1","addr":"172.21.15.157:6815","nonce":115856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":115856},{"type":"v1","addr":"172.21.15.157:6813","nonce":115856}]},"public_addr":"172.21.15.157:6809/115856","cluster_addr":"172.21.15.157:6811/115856","heartbeat_back_addr":"172.21.15.157:6815/115856","heartbeat_front_addr":"172.21.15.157:6813/115856","state":["exists","up"]},{"osd":5,"uuid":"9bfeeafc-a943-41c9-8552-b83a7945a936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":115886},{"type":"v1","addr":"172.21.15.157:6817","nonce":115886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":115886},{"type":"v1","addr":"172.21.15.157:6819","nonce":115886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":115886},{"type":"v1","addr":"172.21.15.157:6823","nonce":115886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":115886},{"type":"v1","addr":"172.21.15.157:6821","nonce":115886}]},"public_addr":"172.21.15.157:6817/115886","cluster_addr":"172.21.15.157:6819/115886","heartbeat_back_addr":"172.21.15.157:6823/115886","heartbeat_front_addr":"172.21.15.157:6821/115886","state":["exists","up"]},{"osd":6,"uuid":"8e859338-35cf-4c43-b0d0-1fede34770d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":115887},{"type":"v1","addr":"172.21.15.157:6825","nonce":115887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":115887},{"type":"v1","addr":"172.21.15.157:6827","nonce":115887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":115887},{"type":"v1","addr":"172.21.15.157:6831","nonce":115887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":115887},{"type":"v1","addr":"172.21.15.157:6829","nonce":115887}]},"public_addr":"172.21.15.157:6825/115887","cluster_addr":"172.21.15.157:6827/115887","heartbeat_back_addr":"172.21.15.157:6831/115887","heartbeat_front_addr":"172.21.15.157:6829/115887","state":["exists","up"]},{"osd":7,"uuid":"ff1fa622-2c14-4469-ae34-10d233bfabd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":115857},{"type":"v1","addr":"172.21.15.157:6801","nonce":115857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":115857},{"type":"v1","addr":"172.21.15.157:6803","nonce":115857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":115857},{"type":"v1","addr":"172.21.15.157:6807","nonce":115857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":115857},{"type":"v1","addr":"172.21.15.157:6805","nonce":115857}]},"public_addr":"172.21.15.157:6801/115857","cluster_addr":"172.21.15.157:6803/115857","heartbeat_back_addr":"172.21.15.157:6807/115857","heartbeat_front_addr":"172.21.15.157:6805/115857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:05.663174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:03.039433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:33:58.914018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:34:28.766204+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:21.668402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:10.540245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:13.728841+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:35:08.575683+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:35:23.847 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T02:35:23.847 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:23.998 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:24.010 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T02:35:24.321 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T02:35:24.332 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T02:35:24.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:24.476 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:24.477 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:24.477 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:24.488 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T02:35:24.801 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T02:35:24.812 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T02:35:24.812 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:24.954 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:24.954 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:24.954 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:24.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T02:35:25.278 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T02:35:25.289 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T02:35:25.289 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:25.430 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:25.431 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:25.431 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:25.442 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T02:35:25.756 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T02:35:25.767 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T02:35:25.767 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:25.910 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:25.910 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:25.910 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:25.922 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T02:35:26.231 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T02:35:26.243 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T02:35:26.243 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:26.385 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:26.386 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:26.386 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:26.397 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T02:35:26.713 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T02:35:26.724 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T02:35:26.724 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:26.867 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:26.867 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:26.868 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:26.879 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T02:35:27.191 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T02:35:27.202 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T02:35:27.202 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:35:27.344 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-09-30T02:35:27.344 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:35:27.344 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-09-30T02:35:27.356 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T02:35:27.671 INFO:teuthology.orchestra.run.smithi115.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T02:35:27.683 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:35:27.992 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:27.993 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:35:28.004 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T02:35:26.904028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":47388,"kb_used_data":4764,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":754927332,"statfs":{"total":773094113280,"available":773045587968,"internally_reserved":0,"allocated":4878336,"data_stored":2612422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424741+0000","last_change":"2023-09-30T02:35:07.048881+0000","last_active":"2023-09-30T02:35:11.424741+0000","last_peered":"2023-09-30T02:35:11.424741+0000","last_clean":"2023-09-30T02:35:11.424741+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:35:11.424741+0000","last_undegraded":"2023-09-30T02:35:11.424741+0000","last_fullsized":"2023-09-30T02:35:11.424741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:07.048791+0000","last_clean_scrub_stamp":"2023-09-30T02:35:07.048791+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:30.482046+0000","scrub_duration":0.30349070500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.424312+0000","last_change":"2023-09-30T02:34:14.080539+0000","last_active":"2023-09-30T02:35:11.424312+0000","last_peered":"2023-09-30T02:35:11.424312+0000","last_clean":"2023-09-30T02:35:11.424312+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:35:11.424312+0000","last_undegraded":"2023-09-30T02:35:11.424312+0000","last_fullsized":"2023-09-30T02:35:11.424312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.080455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.080455+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:41.316039+0000","scrub_duration":0.50328109300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-09-30T02:35:15.121329+0000","last_change":"2023-09-30T02:34:14.489887+0000","last_active":"2023-09-30T02:35:15.121329+0000","last_peered":"2023-09-30T02:35:15.121329+0000","last_clean":"2023-09-30T02:35:15.121329+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:35:15.121329+0000","last_undegraded":"2023-09-30T02:35:15.121329+0000","last_fullsized":"2023-09-30T02:35:15.121329+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:14.489712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:32:45.821578+0000","last_clean_scrub_stamp":"2023-09-30T02:34:14.489712+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:39.582994+0000","scrub_duration":0.502725164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-09-30T02:35:21.182112+0000","last_change":"2023-09-30T02:35:21.182068+0000","last_active":"2023-09-30T02:35:21.182112+0000","last_peered":"2023-09-30T02:35:21.182112+0000","last_clean":"2023-09-30T02:35:21.182112+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:35:21.182112+0000","last_undegraded":"2023-09-30T02:35:21.182112+0000","last_fullsized":"2023-09-30T02:35:21.182112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:21.181965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:35:21.181965+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:50.039912+0000","scrub_duration":0.20189895899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":217,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-09-30T02:35:15.221433+0000","last_change":"2023-09-30T02:34:01.305491+0000","last_active":"2023-09-30T02:35:15.221433+0000","last_peered":"2023-09-30T02:35:15.221433+0000","last_clean":"2023-09-30T02:35:15.221433+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:35:15.221433+0000","last_undegraded":"2023-09-30T02:35:15.221433+0000","last_fullsized":"2023-09-30T02:35:15.221433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2023-09-30T02:34:01.305372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:13.863889+0000","last_clean_scrub_stamp":"2023-09-30T02:34:01.305372+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:27.415083+0000","scrub_duration":0.40451995600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.866877+0000","last_change":"2023-09-30T02:34:27.832803+0000","last_active":"2023-09-30T02:35:12.866877+0000","last_peered":"2023-09-30T02:35:12.866877+0000","last_clean":"2023-09-30T02:35:12.866877+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:35:12.866877+0000","last_undegraded":"2023-09-30T02:35:12.866877+0000","last_fullsized":"2023-09-30T02:35:12.866877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:27.832676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:27.832676+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:35.373737+0000","scrub_duration":0.20230350499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:11.909951+0000","last_change":"2023-09-30T02:34:20.302744+0000","last_active":"2023-09-30T02:35:11.909951+0000","last_peered":"2023-09-30T02:35:11.909951+0000","last_clean":"2023-09-30T02:35:11.909951+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:35:11.909951+0000","last_undegraded":"2023-09-30T02:35:11.909951+0000","last_fullsized":"2023-09-30T02:35:11.909951+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:34:20.302648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:34:20.302648+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:24.481708+0000","scrub_duration":0.30196727200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:12.666476+0000","last_change":"2023-09-30T02:33:54.052852+0000","last_active":"2023-09-30T02:35:12.666476+0000","last_peered":"2023-09-30T02:35:12.666476+0000","last_clean":"2023-09-30T02:35:12.666476+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:35:12.666476+0000","last_undegraded":"2023-09-30T02:35:12.666476+0000","last_fullsized":"2023-09-30T02:35:12.666476+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:33:54.052650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:30:14.869536+0000","last_clean_scrub_stamp":"2023-09-30T02:33:54.052650+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:35:21.704144+0000","scrub_duration":0.50227995400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-09-30T02:35:14.710861+0000","last_change":"2023-09-30T02:35:14.710829+0000","last_active":"2023-09-30T02:35:14.710861+0000","last_peered":"2023-09-30T02:35:14.710861+0000","last_clean":"2023-09-30T02:35:14.710861+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:35:14.710861+0000","last_undegraded":"2023-09-30T02:35:14.710861+0000","last_fullsized":"2023-09-30T02:35:14.710861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:14.710754+0000","last_clean_scrub_stamp":"2023-09-30T02:35:14.710754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:41.843718+0000","scrub_duration":0.20152634699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6696,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365144,"statfs":{"total":96636764160,"available":96629907456,"internally_reserved":0,"allocated":1015808,"data_stored":727736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58599999999999997,"15min":0.67600000000000005},"min":{"1min":0.35199999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69699999999999995,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.80700000000000005,"15min":1.454},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81000000000000005,"5min":22.445,"15min":22.445},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.86299999999999999,"15min":1.6739999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77800000000000002,"5min":22.483000000000001,"15min":22.483000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77800000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.67600000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":3.9870000000000001,"15min":3.9870000000000001},"last":0.626},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.80100000000000005,"15min":1.365},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":22.547999999999998,"15min":22.547999999999998},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59899999999999998,"15min":0.68400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.749,"5min":3.9079999999999999,"15min":3.9079999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.91900000000000004,"15min":1.7549999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76400000000000001,"5min":22.515000000000001,"15min":22.515000000000001},"last":0.63800000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66700000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49299999999999999,"15min":0.52000000000000002},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.59299999999999997}]},{"osd":5,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.71699999999999997,"15min":1.2949999999999999},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72299999999999998,"5min":22.398,"15min":22.398},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77500000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.82799999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.64900000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55400000000000005,"15min":0.51400000000000001},"min":{"1min":0.42899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70199999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.302,"15min":6.96},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71999999999999997,"5min":175.99700000000001,"15min":175.99700000000001},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.48399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66000000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.504,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.55500000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.61599999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.753,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57999999999999996}]},{"osd":4,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.2889999999999999,"15min":3.2799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":57.932000000000002,"15min":57.932000000000002},"last":0.629},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.355,"15min":3.335},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":57.914000000000001,"15min":57.914000000000001},"last":0.65100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5728,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366112,"statfs":{"total":96636764160,"available":96630898688,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.372,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.318},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56499999999999995,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.35599999999999998},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42899999999999999,"15min":0.42399999999999999},"min":{"1min":0.30399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64400000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.996,"15min":0.996},"last":0.499},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.505,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.72999999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.45100000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.748,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84599999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.53900000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.621},"min":{"1min":0.49399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81399999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.629,"15min":0.627},"min":{"1min":0.45900000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81100000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6680,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365160,"statfs":{"total":96636764160,"available":96629923840,"internally_reserved":0,"allocated":999424,"data_stored":721952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.42699999999999999},"min":{"1min":0.23499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.55500000000000005,"5min":1.629,"15min":1.629},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.216,"15min":0.216},"max":{"1min":0.621,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.222}]},{"osd":2,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.40699999999999997,"15min":0.44500000000000001},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.61599999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.32000000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.85399999999999998,"15min":1.4019999999999999},"min":{"1min":0.251,"5min":0.22,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.251},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.49199999999999999},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.373}]},{"osd":4,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.63},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78500000000000003,"5min":1.361,"15min":1.361},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.0660000000000001,"15min":1.6100000000000001},"min":{"1min":0.48699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.874,"5min":26.218,"15min":26.218},"last":0.71099999999999997}]},{"osd":5,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.0960000000000001,"15min":1.6439999999999999},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":26.247,"15min":26.247},"last":0.751},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.52300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96399999999999997,"5min":1.583,"15min":1.583},"last":0.55500000000000005}]},{"osd":6,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64000000000000001,"15min":0.67900000000000005},"min":{"1min":0.46300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92400000000000004,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":1.409,"15min":1.409},"last":0.72099999999999997}]},{"osd":7,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64600000000000002,"15min":0.67200000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.92100000000000004,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.095,"15min":1.661},"min":{"1min":0.44700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.93999999999999995,"5min":26.271999999999998,"15min":26.271999999999998},"last":0.73499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":475136,"data_stored":193646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":1.268,"15min":3.6539999999999999},"min":{"1min":0.221,"5min":0.217,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":81.966999999999999,"15min":81.966999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.40000000000000002,"15min":0.45600000000000002},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.53600000000000003,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.48799999999999999,"15min":0.77000000000000002},"min":{"1min":0.26500000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51000000000000001,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.56299999999999994,"15min":0.99299999999999999},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54100000000000004,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.51800000000000002,"15min":0.85399999999999998},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.51300000000000001,"5min":12.164,"15min":12.164},"last":0.308},{"interface":"front","average":{"1min":0.437,"5min":0.52700000000000002,"15min":0.77900000000000003},"min":{"1min":0.32600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.62,"5min":7.101,"15min":7.101},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.69599999999999995},"min":{"1min":0.51100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71899999999999997,"5min":2.7639999999999998,"15min":2.7639999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.498,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.88900000000000001,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.83299999999999996,"15min":1.3400000000000001},"min":{"1min":0.52900000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":12.07,"15min":12.07},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":1.0580000000000001},"min":{"1min":0.57199999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98999999999999999,"5min":10.353999999999999,"15min":10.353999999999999},"last":0.621}]},{"osd":6,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.46300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97299999999999998,"5min":1.103,"15min":1.103},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.751,"15min":1.083},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86699999999999999,"5min":12.135,"15min":12.135},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68700000000000006,"15min":0.79900000000000004},"min":{"1min":0.48199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84399999999999997,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.79000000000000004,"15min":1.226},"min":{"1min":0.52500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94899999999999995,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.59499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5664,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366176,"statfs":{"total":96636764160,"available":96630964224,"internally_reserved":0,"allocated":483328,"data_stored":194218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.504,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.372},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48099999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.33300000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.36599999999999999,"15min":0.35999999999999999},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.32000000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.31900000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53200000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.40999999999999998}]},{"osd":4,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.87,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61099999999999999,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80600000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65100000000000002,"15min":0.65600000000000003},"min":{"1min":0.46600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82999999999999996,"5min":1.006,"15min":1.006},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.2629999999999999,"15min":0.82899999999999996},"min":{"1min":0.45000000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.747,"5min":65.600999999999999,"15min":65.600999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65100000000000002,"15min":0.66900000000000004},"min":{"1min":0.499,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97899999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52800000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77300000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.84699999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5660,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366180,"statfs":{"total":96636764160,"available":96630968320,"internally_reserved":0,"allocated":479232,"data_stored":193932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66800000000000004,"5min":0.754,"15min":0.754},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.38300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.55600000000000005},"min":{"1min":0.437,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.42999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72699999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.51100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82899999999999996,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.35899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.496,"15min":0.47599999999999998},"min":{"1min":0.377,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.433}]},{"osd":6,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75800000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56000000000000005}]},{"osd":7,"last update":"Sat Sep 30 02:34:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.56699999999999995},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89600000000000002,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":460,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":471040,"data_stored":193360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":4.883,"15min":9.2129999999999992},"min":{"1min":0.377,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73599999999999999,"5min":192.72,"15min":192.72},"last":0.57199999999999995}]},{"osd":1,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.623,"15min":0.68600000000000005},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79800000000000004,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.502},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.66400000000000003,"15min":0.75600000000000001},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81000000000000005,"5min":4.5,"15min":4.5},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.61299999999999999,"15min":0.65400000000000003},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.67700000000000005},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72299999999999998,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66100000000000003,"15min":0.74099999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.73699999999999999,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.67500000000000004,"15min":0.78100000000000003},"min":{"1min":0.43099999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.75600000000000001,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70799999999999996,"5min":2.774,"15min":2.774},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.59799999999999998},"min":{"1min":0.35599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73899999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.38100000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58099999999999996,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64700000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60899999999999999,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71399999999999997,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73499999999999999,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.442},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.4059999999999999,"15min":2.2050000000000001},"min":{"1min":0.502,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69199999999999995,"5min":33.798000000000002,"15min":33.798000000000002},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:35:28.009 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T02:35:07.048791+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.009 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T02:34:14.080455+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=34, tm_sec=14, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.009 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T02:34:14.489712+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=34, tm_sec=14, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.009 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T02:35:21.181965+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=21, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.010 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T02:34:01.305372+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=34, tm_sec=1, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.010 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T02:34:27.832676+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=34, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.010 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T02:34:20.302648+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=34, tm_sec=20, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.010 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T02:33:54.052650+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=33, tm_sec=54, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.010 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T02:35:14.710754+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=14, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=35, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:35:28.011 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T02:35:48.012 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:35:48.338 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-09-30T02:35:48.339 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-09-30T02:35:48.349 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2023-09-30T02:35:48.240786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":47732,"kb_used_data":4916,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":754926988,"statfs":{"total":773094113280,"available":773045235712,"internally_reserved":0,"allocated":5033984,"data_stored":2692140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.334626"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T02:35:31.432400+0000","last_change":"2023-09-30T02:35:31.432352+0000","last_active":"2023-09-30T02:35:31.432400+0000","last_peered":"2023-09-30T02:35:31.432400+0000","last_clean":"2023-09-30T02:35:31.432400+0000","last_became_active":"2023-09-30T02:30:15.881330+0000","last_became_peered":"2023-09-30T02:30:15.881330+0000","last_unstale":"2023-09-30T02:35:31.432400+0000","last_undegraded":"2023-09-30T02:35:31.432400+0000","last_fullsized":"2023-09-30T02:35:31.432400+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:31.432243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:31.432243+0000","last_clean_scrub_stamp":"2023-09-30T02:35:31.432243+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:58.200138+0000","scrub_duration":0.502541762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":78,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T02:35:30.632693+0000","last_change":"2023-09-30T02:35:30.632666+0000","last_active":"2023-09-30T02:35:30.632693+0000","last_peered":"2023-09-30T02:35:30.632693+0000","last_clean":"2023-09-30T02:35:30.632693+0000","last_became_active":"2023-09-30T02:30:16.076797+0000","last_became_peered":"2023-09-30T02:30:16.076797+0000","last_unstale":"2023-09-30T02:35:30.632693+0000","last_undegraded":"2023-09-30T02:35:30.632693+0000","last_fullsized":"2023-09-30T02:35:30.632693+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:30.632563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:30.632563+0000","last_clean_scrub_stamp":"2023-09-30T02:35:30.632563+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:40.860277+0000","scrub_duration":0.001693396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":78,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T02:35:39.101953+0000","last_change":"2023-09-30T02:35:39.101924+0000","last_active":"2023-09-30T02:35:39.101953+0000","last_peered":"2023-09-30T02:35:39.101953+0000","last_clean":"2023-09-30T02:35:39.101953+0000","last_became_active":"2023-09-30T02:30:15.910503+0000","last_became_peered":"2023-09-30T02:30:15.910503+0000","last_unstale":"2023-09-30T02:35:39.101953+0000","last_undegraded":"2023-09-30T02:35:39.101953+0000","last_fullsized":"2023-09-30T02:35:39.101953+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:39.101813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:39.101813+0000","last_clean_scrub_stamp":"2023-09-30T02:35:39.101813+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:37:05.316245+0000","scrub_duration":0.102327736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T02:35:32.198105+0000","last_change":"2023-09-30T02:35:32.198068+0000","last_active":"2023-09-30T02:35:32.198105+0000","last_peered":"2023-09-30T02:35:32.198105+0000","last_clean":"2023-09-30T02:35:32.198105+0000","last_became_active":"2023-09-30T02:30:15.881231+0000","last_became_peered":"2023-09-30T02:30:15.881231+0000","last_unstale":"2023-09-30T02:35:32.198105+0000","last_undegraded":"2023-09-30T02:35:32.198105+0000","last_fullsized":"2023-09-30T02:35:32.198105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:32.197963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:32.197963+0000","last_clean_scrub_stamp":"2023-09-30T02:35:32.197963+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:35.148335+0000","scrub_duration":0.20212116299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"44'119","reported_seq":231,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-09-30T02:35:29.436061+0000","last_change":"2023-09-30T02:35:29.435950+0000","last_active":"2023-09-30T02:35:29.436061+0000","last_peered":"2023-09-30T02:35:29.436061+0000","last_clean":"2023-09-30T02:35:29.436061+0000","last_became_active":"2023-09-30T02:30:14.879784+0000","last_became_peered":"2023-09-30T02:30:14.879784+0000","last_unstale":"2023-09-30T02:35:29.436061+0000","last_undegraded":"2023-09-30T02:35:29.436061+0000","last_fullsized":"2023-09-30T02:35:29.436061+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'119","last_scrub_stamp":"2023-09-30T02:35:29.435754+0000","last_deep_scrub":"44'119","last_deep_scrub_stamp":"2023-09-30T02:35:29.435754+0000","last_clean_scrub_stamp":"2023-09-30T02:35:29.435754+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:40.488568+0000","scrub_duration":0.40536624500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-09-30T02:35:29.080498+0000","last_change":"2023-09-30T02:35:29.080446+0000","last_active":"2023-09-30T02:35:29.080498+0000","last_peered":"2023-09-30T02:35:29.080498+0000","last_clean":"2023-09-30T02:35:29.080498+0000","last_became_active":"2023-09-30T02:30:15.980284+0000","last_became_peered":"2023-09-30T02:30:15.980284+0000","last_unstale":"2023-09-30T02:35:29.080498+0000","last_undegraded":"2023-09-30T02:35:29.080498+0000","last_fullsized":"2023-09-30T02:35:29.080498+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:29.080335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:29.080335+0000","last_clean_scrub_stamp":"2023-09-30T02:35:29.080335+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:58.134019+0000","scrub_duration":0.40248992700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-09-30T02:35:27.207840+0000","last_change":"2023-09-30T02:35:27.207780+0000","last_active":"2023-09-30T02:35:27.207840+0000","last_peered":"2023-09-30T02:35:27.207840+0000","last_clean":"2023-09-30T02:35:27.207840+0000","last_became_active":"2023-09-30T02:30:16.013257+0000","last_became_peered":"2023-09-30T02:30:16.013257+0000","last_unstale":"2023-09-30T02:35:27.207840+0000","last_undegraded":"2023-09-30T02:35:27.207840+0000","last_fullsized":"2023-09-30T02:35:27.207840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:27.207679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:27.207679+0000","last_clean_scrub_stamp":"2023-09-30T02:35:27.207679+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:54.700111+0000","scrub_duration":0.40183395799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-09-30T02:35:29.734835+0000","last_change":"2023-09-30T02:35:29.734775+0000","last_active":"2023-09-30T02:35:29.734835+0000","last_peered":"2023-09-30T02:35:29.734835+0000","last_clean":"2023-09-30T02:35:29.734835+0000","last_became_active":"2023-09-30T02:30:15.981147+0000","last_became_peered":"2023-09-30T02:30:15.981147+0000","last_unstale":"2023-09-30T02:35:29.734835+0000","last_undegraded":"2023-09-30T02:35:29.734835+0000","last_fullsized":"2023-09-30T02:35:29.734835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:29.734669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:29.734669+0000","last_clean_scrub_stamp":"2023-09-30T02:35:29.734669+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:45.085116+0000","scrub_duration":0.102428071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-09-30T02:35:27.884187+0000","last_change":"2023-09-30T02:35:27.884129+0000","last_active":"2023-09-30T02:35:27.884187+0000","last_peered":"2023-09-30T02:35:27.884187+0000","last_clean":"2023-09-30T02:35:27.884187+0000","last_became_active":"2023-09-30T02:30:15.976707+0000","last_became_peered":"2023-09-30T02:30:15.976707+0000","last_unstale":"2023-09-30T02:35:27.884187+0000","last_undegraded":"2023-09-30T02:35:27.884187+0000","last_fullsized":"2023-09-30T02:35:27.884187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:35:27.884037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:35:27.884037+0000","last_clean_scrub_stamp":"2023-09-30T02:35:27.884037+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T02:36:38.479949+0000","scrub_duration":0.30191008899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6708,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365132,"statfs":{"total":96636764160,"available":96629895168,"internally_reserved":0,"allocated":1028096,"data_stored":733184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58599999999999997,"15min":0.67600000000000005},"min":{"1min":0.35199999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69699999999999995,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.80700000000000005,"15min":1.454},"min":{"1min":0.36899999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81000000000000005,"5min":22.445,"15min":22.445},"last":0.74299999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.86299999999999999,"15min":1.6739999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77800000000000002,"5min":22.483000000000001,"15min":22.483000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77800000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65700000000000003}]},{"osd":2,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.67600000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79500000000000004,"5min":3.9870000000000001,"15min":3.9870000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.80100000000000005,"15min":1.365},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.745,"5min":22.547999999999998,"15min":22.547999999999998},"last":0.67700000000000005}]},{"osd":3,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59899999999999998,"15min":0.68400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.749,"5min":3.9079999999999999,"15min":3.9079999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.91900000000000004,"15min":1.7549999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76400000000000001,"5min":22.515000000000001,"15min":22.515000000000001},"last":0.71499999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66700000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49299999999999999,"15min":0.52000000000000002},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.71699999999999997,"15min":1.2949999999999999},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72299999999999998,"5min":22.398,"15min":22.398},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77500000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Sep 30 02:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.55900000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.82799999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5736,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366104,"statfs":{"total":96636764160,"available":96630890496,"internally_reserved":0,"allocated":491520,"data_stored":203970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55400000000000005,"15min":0.51400000000000001},"min":{"1min":0.42899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70199999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.302,"15min":6.96},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71999999999999997,"5min":175.99700000000001,"15min":175.99700000000001},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.53900000000000003},"min":{"1min":0.48399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66000000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.504,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.61699999999999999,"15min":0.59399999999999997},"min":{"1min":0.55500000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.625}]},{"osd":3,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.753,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58599999999999997}]},{"osd":4,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54800000000000004}]},{"osd":5,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.441},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64200000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:35:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.2889999999999999,"15min":3.2799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73499999999999999,"5min":57.932000000000002,"15min":57.932000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.355,"15min":3.335},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":57.914000000000001,"15min":57.914000000000001},"last":0.66300000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5748,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366092,"statfs":{"total":96636764160,"available":96630878208,"internally_reserved":0,"allocated":503808,"data_stored":204828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36299999999999999,"15min":0.37},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.435,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40300000000000002},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.45700000000000002,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.378}]},{"osd":2,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.36099999999999999,"15min":0.35499999999999998},"min":{"1min":0.224,"5min":0.189,"15min":0.189},"max":{"1min":0.51100000000000001,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.38,"15min":0.37},"min":{"1min":0.27100000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.42999999999999999,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.42299999999999999},"min":{"1min":0.29599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.47399999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.41199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.48999999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.436}]},{"osd":4,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58399999999999996,"15min":0.57099999999999995},"min":{"1min":0.441,"5min":0.38,"15min":0.38},"max":{"1min":0.71299999999999997,"5min":0.996,"15min":0.996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80200000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.45000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82699999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.49399999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.42499999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.628,"15min":0.62},"min":{"1min":0.46200000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86899999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.73399999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.625,"15min":0.625},"min":{"1min":0.47899999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71899999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57899999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6700,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365140,"statfs":{"total":96636764160,"available":96629903360,"internally_reserved":0,"allocated":1019904,"data_stored":732562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.42699999999999999},"min":{"1min":0.23499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.55500000000000005,"5min":1.629,"15min":1.629},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.216,"15min":0.216},"max":{"1min":0.621,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.40699999999999997,"15min":0.44500000000000001},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.61599999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.85399999999999998,"15min":1.4019999999999999},"min":{"1min":0.251,"5min":0.22,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.49199999999999999},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67000000000000004,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60399999999999998,"15min":0.63},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78500000000000003,"5min":1.361,"15min":1.361},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.0660000000000001,"15min":1.6100000000000001},"min":{"1min":0.48699999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.874,"5min":26.218,"15min":26.218},"last":0.502}]},{"osd":5,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.0960000000000001,"15min":1.6439999999999999},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.876,"5min":26.247,"15min":26.247},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.52300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96399999999999997,"5min":1.583,"15min":1.583},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64000000000000001,"15min":0.67900000000000005},"min":{"1min":0.46300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92400000000000004,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":1.409,"15min":1.409},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Sep 30 02:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64600000000000002,"15min":0.67200000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.92100000000000004,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.095,"15min":1.661},"min":{"1min":0.44700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.93999999999999995,"5min":26.271999999999998,"15min":26.271999999999998},"last":0.59399999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5740,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366100,"statfs":{"total":96636764160,"available":96630886400,"internally_reserved":0,"allocated":495616,"data_stored":204256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":1.268,"15min":3.6539999999999999},"min":{"1min":0.221,"5min":0.217,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":81.966999999999999,"15min":81.966999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.40000000000000002,"15min":0.45600000000000002},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.53600000000000003,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.48799999999999999,"15min":0.77000000000000002},"min":{"1min":0.26500000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51000000000000001,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":0.371},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.56299999999999994,"15min":0.99299999999999999},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54100000000000004,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.34499999999999997}]},{"osd":3,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.51800000000000002,"15min":0.85399999999999998},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.51300000000000001,"5min":12.164,"15min":12.164},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.52700000000000002,"15min":0.77900000000000003},"min":{"1min":0.32600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.62,"5min":7.101,"15min":7.101},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.69599999999999995},"min":{"1min":0.51100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71899999999999997,"5min":2.7639999999999998,"15min":2.7639999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.628,"15min":0.64400000000000002},"min":{"1min":0.498,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.88900000000000001,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.83299999999999996,"15min":1.3400000000000001},"min":{"1min":0.52900000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":12.07,"15min":12.07},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77500000000000002,"15min":1.0580000000000001},"min":{"1min":0.57199999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98999999999999999,"5min":10.353999999999999,"15min":10.353999999999999},"last":0.621}]},{"osd":6,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.46300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97299999999999998,"5min":1.103,"15min":1.103},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.751,"15min":1.083},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86699999999999999,"5min":12.135,"15min":12.135},"last":0.58199999999999996}]},{"osd":7,"last update":"Sat Sep 30 02:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.68700000000000006,"15min":0.79900000000000004},"min":{"1min":0.48199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84399999999999997,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.79000000000000004,"15min":1.226},"min":{"1min":0.52500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94899999999999995,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.63800000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5684,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366156,"statfs":{"total":96636764160,"available":96630943744,"internally_reserved":0,"allocated":503808,"data_stored":204828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.504,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48099999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.36599999999999999,"15min":0.35999999999999999},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.55300000000000005,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.377},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39200000000000002,"15min":0.40200000000000002},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.31900000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53200000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.87,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.505},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61099999999999999,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80600000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60499999999999998}]},{"osd":5,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65100000000000002,"15min":0.65600000000000003},"min":{"1min":0.46600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82999999999999996,"5min":1.006,"15min":1.006},"last":0.54100000000000004}]},{"osd":6,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.2629999999999999,"15min":0.82899999999999996},"min":{"1min":0.45000000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.747,"5min":65.600999999999999,"15min":65.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65100000000000002,"15min":0.66900000000000004},"min":{"1min":0.499,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97899999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.65200000000000002},"min":{"1min":0.45100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77300000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.439},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.84699999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.63300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5680,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94366160,"statfs":{"total":96636764160,"available":96630947840,"internally_reserved":0,"allocated":499712,"data_stored":204542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.34300000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.749,"5min":0.754,"15min":0.754},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.35899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78500000000000003,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.376,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71899999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.55600000000000005},"min":{"1min":0.38,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77100000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55500000000000005,"15min":0.55200000000000005},"min":{"1min":0.44800000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.56699999999999995}]},{"osd":3,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.505,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79700000000000004,"5min":0.87,"15min":0.87},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.46800000000000003,"15min":0.47799999999999998},"min":{"1min":0.37,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.48199999999999998},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.86199999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.307}]},{"osd":6,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49399999999999999,"15min":0.48799999999999999},"min":{"1min":0.39700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76500000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.53800000000000003},"min":{"1min":0.42799999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.38400000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:35:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.58799999999999997,"15min":0.56999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93500000000000005,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.34499999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5736,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366104,"statfs":{"total":96636764160,"available":96630890496,"internally_reserved":0,"allocated":491520,"data_stored":203970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58799999999999997,"15min":0.63},"min":{"1min":0.374,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69399999999999995,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":2.7229999999999999,"15min":8.4930000000000003},"min":{"1min":0.44900000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81200000000000006,"5min":192.72,"15min":192.72},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58899999999999997,"15min":0.67500000000000004},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68999999999999995,"5min":3.7170000000000001,"15min":3.7170000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61699999999999999,"15min":0.73999999999999999},"min":{"1min":0.42099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":4.5,"15min":4.5},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58799999999999997,"15min":0.64600000000000002},"min":{"1min":0.41399999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61199999999999999,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71699999999999997,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.61699999999999999,"15min":0.72699999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69499999999999995,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.76200000000000001},"min":{"1min":0.46899999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.626,"5min":5.2560000000000002,"15min":5.2560000000000002},"last":0.68600000000000005}]},{"osd":4,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50700000000000001,"15min":0.57099999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66200000000000003,"5min":2.774,"15min":2.774},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52900000000000003,"15min":0.58899999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54600000000000004,"15min":0.629},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70199999999999996,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58299999999999996,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73399999999999999,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.74199999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:35:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.63800000000000001},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":3.1869999999999998,"15min":3.1869999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.99399999999999999,"15min":2.0680000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72999999999999998,"5min":33.798000000000002,"15min":33.798000000000002},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:35:48.352 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T02:35:48.682 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:35:48.683 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T02:35:48.684 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T02:35:48.684 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T02:35:48.684 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:48.720 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T02:35:48.720 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T02:35:48.720 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:48.790 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T02:35:48.790 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T02:35:48.790 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:48.858 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T02:35:48.858 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T02:35:48.858 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:48.921 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T02:35:48.921 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T02:35:48.921 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:48.964 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T02:35:48.964 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T02:35:48.964 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.029 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T02:35:49.030 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T02:35:49.030 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.096 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T02:35:49.096 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T02:35:49.096 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.161 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T02:35:49.162 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T02:35:49.162 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T02:35:49.162 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.185 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T02:35:49.185 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T02:35:49.185 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.226 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T02:35:49.226 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T02:35:49.226 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T02:35:49.226 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.236 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T02:35:49.237 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T02:35:49.237 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.287 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T02:35:49.287 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T02:35:49.287 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:35:49.298 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T02:35:49.298 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T02:35:49.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T02:35:49.359 INFO:teuthology.orchestra.run.smithi115.stdout:1696041008.8641481 mgr.y (mgr.4110) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:35:49.359 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T02:35:49.359 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T02:35:49.432 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T02:35:49.503 INFO:teuthology.orchestra.run.smithi115.stdout:1696041008.8641481 mgr.y (mgr.4110) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:35:49.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:35:49.503 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T02:35:49.699 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:35:49.699 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T02:35:49.774 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:35:49.774 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T02:35:49.876 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:35:49.876 DEBUG:teuthology.orchestra.run.smithi115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T02:35:49.959 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:35:49.959 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T02:35:50.146 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:35:50.147 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T02:35:50.249 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:35:50.249 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T02:35:50.313 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:35:50.313 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T02:35:50.379 INFO:tasks.ceph:Archiving mon data... 2023-09-30T02:35:50.380 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/data/mon.a.tgz 2023-09-30T02:35:50.381 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-09-30T02:35:50.399 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.YVzKzg0OxY 2023-09-30T02:35:50.400 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YVzKzg0OxY 2023-09-30T02:35:50.542 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.YVzKzg0OxY 2023-09-30T02:35:50.629 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.YVzKzg0OxY is 298KB 2023-09-30T02:35:50.723 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.YVzKzg0OxY 2023-09-30T02:35:50.741 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/data/mon.c.tgz 2023-09-30T02:35:50.741 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-09-30T02:35:50.798 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.sCp9qkKM0i 2023-09-30T02:35:50.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.sCp9qkKM0i 2023-09-30T02:35:50.914 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.sCp9qkKM0i 2023-09-30T02:35:51.001 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.sCp9qkKM0i is 298KB 2023-09-30T02:35:51.096 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.sCp9qkKM0i 2023-09-30T02:35:51.113 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/data/mon.b.tgz 2023-09-30T02:35:51.114 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-09-30T02:35:51.136 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.xXwY8XU6By 2023-09-30T02:35:51.137 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.xXwY8XU6By 2023-09-30T02:35:51.271 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.xXwY8XU6By 2023-09-30T02:35:51.360 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.xXwY8XU6By is 298KB 2023-09-30T02:35:51.455 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.xXwY8XU6By 2023-09-30T02:35:51.473 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T02:35:51.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:35:51.475 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:35:51.539 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T02:35:51.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:35:51.545 DEBUG:teuthology.orchestra.run.smithi157:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:35:51.572 INFO:teuthology.orchestra.run.smithi115.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T02:35:51.607 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T02:35:51.613 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T02:35:51.614 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/remote/smithi115/crash 2023-09-30T02:35:51.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:35:51.649 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/remote/smithi157/crash 2023-09-30T02:35:51.650 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:35:51.680 INFO:tasks.ceph:Compressing logs... 2023-09-30T02:35:51.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:35:51.694 DEBUG:teuthology.orchestra.run.smithi157:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:35:52.663 INFO:tasks.ceph:Archiving logs... 2023-09-30T02:35:52.664 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/remote/smithi115/log 2023-09-30T02:35:52.665 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:35:52.930 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/remote/smithi157/log 2023-09-30T02:35:52.931 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:35:53.060 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:35:53.125 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=94a72dc090ff43c4af7ff6459c975596 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:35:53.127 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T02:35:53.146 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T02:35:53.184 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T02:35:53.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:35:53.196 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:35:53.274 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T02:35:53.284 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T02:35:53.284 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:35:53.287 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:35:53.302 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-09-30T02:35:53.305 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-09-30T02:35:53.306 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:35:53.306 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-09-30T02:35:53.307 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 -262us[ -251us] +/- 70ms 2023-09-30T02:35:53.307 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -215us[ -205us] +/- 52ms 2023-09-30T02:35:53.307 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +589us[ +599us] +/- 73ms 2023-09-30T02:35:53.307 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:35:53.309 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:35:53.310 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-09-30T02:35:53.310 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 -236us[ -236us] +/- 70ms 2023-09-30T02:35:53.310 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -207us[ -186us] +/- 52ms 2023-09-30T02:35:53.310 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +610us[ +610us] +/- 73ms 2023-09-30T02:35:53.310 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:35:53.311 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T02:35:53.320 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T02:35:53.321 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T02:35:53.331 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T02:35:53.440 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T02:35:53.533 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:35:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:35:53.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:35:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:35:53.599 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T02:35:53.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T02:35:53.618 INFO:teuthology.task.internal:Duration was 1052.499949 seconds 2023-09-30T02:35:53.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T02:35:53.630 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T02:35:53.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:35:53.632 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:35:53.676 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:35:53.687 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:35:53.986 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T02:35:53.986 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2023-09-30T02:35:53.986 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:35:54.011 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T02:35:54.011 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:35:54.035 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T02:35:54.035 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:35:54.053 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:35:54.114 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T02:35:54.114 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:35:54.133 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:35:54.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T02:35:54.232 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T02:35:54.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:35:54.259 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:35:54.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T02:35:54.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:35:54.302 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:35:54.326 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2023-09-30T02:35:54.354 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2023-09-30T02:35:54.382 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:35:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:35:54.400 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:35:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:35:54.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T02:35:54.449 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T02:35:54.449 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/remote/smithi115 2023-09-30T02:35:54.450 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:35:54.487 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271/remote/smithi157 2023-09-30T02:35:54.488 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:35:54.525 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T02:35:54.525 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:35:54.528 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:35:54.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T02:35:54.592 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T02:35:54.592 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T02:35:54.606 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T02:35:54.607 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:35:54.609 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:35:54.624 INFO:teuthology.orchestra.run.smithi115.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:35 /home/ubuntu/cephtest 2023-09-30T02:35:54.636 INFO:teuthology.orchestra.run.smithi157.stdout: 526080 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:35 /home/ubuntu/cephtest 2023-09-30T02:35:54.638 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T02:35:54.647 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T02:35:54.674 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T02:35:54.698 DEBUG:teuthology.nuke:shortname: smithi115 2023-09-30T02:35:54.699 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:35:54.722 DEBUG:teuthology.nuke:shortname: smithi157 2023-09-30T02:35:54.722 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:35:54.726 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 02:09:32.052576', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYOankZWKJky5kfFILmXO0eIT66frOaGpGYvbpK0S+mqX8fpoL6QLneZhLl/VEuLdRf2xx07HivIisYwtRluaw='} 2023-09-30T02:35:54.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407271', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 02:09:32.051168', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF8TKoW78maGXgk+0aMXgSg1b+cRx/KzgQ1eHyyhdluTQs7ZFD21iieBGxAfqEz43v1NGebJS9xtuog7nnLnJ/g='} 2023-09-30T02:35:54.789 INFO:teuthology.orchestra.console.smithi115:Power off 2023-09-30T02:35:54.789 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:35:54.812 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2023-09-30T02:35:54.812 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:35:54.828 INFO:teuthology.orchestra.console.smithi157:Power off 2023-09-30T02:35:54.828 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:35:54.837 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-09-30T02:35:54.850 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2023-09-30T02:35:54.850 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:35:54.874 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T02:35:58.839 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:35:58.876 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:35:58.949 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-09-30T02:35:58.988 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T02:36:02.953 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:36:02.990 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:36:03.063 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-09-30T02:36:03.100 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-09-30T02:36:03.200 INFO:teuthology.orchestra.console.smithi157:Power off completed 2023-09-30T02:36:07.066 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:36:07.176 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2023-09-30T02:36:07.277 INFO:teuthology.orchestra.console.smithi115:Power off completed 2023-09-30T02:36:07.382 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} duration: 1052.499948978424 failure_reason: '"1696041008.8641481 mgr.y (mgr.4110) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.y: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=94a72dc090ff43c4af7ff6459c975596 status: fail success: false 2023-09-30T02:36:07.382 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T02:36:07.459 INFO:teuthology.run:FAIL