2023-09-30T00:08:47.975 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:08:47.976 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696 --owner scheduled_yuriw@teuthology --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} --verbose -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/orig.config.yaml 2023-09-30T00:08:48.025 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:08:48.116 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} email: yweinste@redhat.com first_in_suite: false job_id: '7406696' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 osd_recovery_max_active: 10 osd_recovery_max_single_start: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd max scrubs: 2 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - must scrub before tier agent can activate sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 min_in: 4 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbLSWpInlss1NhwAmRhZO8PmH0HZUnnVKkzJwG9jwzE9fx6pDf+J6C0HPNEsaoc1Uc10f1MWySqmXKWh7mnFgE= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCw9suGSXpcUQH2x096mCSKwsSx8zzV4kALqY6WlEzmOhUXQvknAtq3ZjgAPq4f1J85RnSx+ZyY6CVcP57h+Fsw= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - sudo ceph osd pool set cache min_write_recency_for_promote 2 - rados: clients: - client.0 objects: 500 op_weights: copy_from: 50 delete: 50 read: 100 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:08:48.116 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:08:48.117 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:08:48.117 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:08:48.118 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:08:48.152 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:08:48.152 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:08:48.152 INFO:teuthology.packaging:ref: None 2023-09-30T00:08:48.152 INFO:teuthology.packaging:tag: None 2023-09-30T00:08:48.153 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:08:48.153 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:08:48.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:08:48.305 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:08:48.305 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:08:48.320 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:08:48.320 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:08:48.337 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:08:48.355 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:08:48.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:08:48.385 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:03:42.587198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbLSWpInlss1NhwAmRhZO8PmH0HZUnnVKkzJwG9jwzE9fx6pDf+J6C0HPNEsaoc1Uc10f1MWySqmXKWh7mnFgE='} 2023-09-30T00:08:48.405 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:03:42.588725', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCw9suGSXpcUQH2x096mCSKwsSx8zzV4kALqY6WlEzmOhUXQvknAtq3ZjgAPq4f1J85RnSx+ZyY6CVcP57h+Fsw='} 2023-09-30T00:08:48.405 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:08:48.411 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T00:08:48.411 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T00:08:48.411 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:08:48.471 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd1a46d8ee0>, signals=[15]) 2023-09-30T00:08:48.472 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:08:48.477 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:08:48.477 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:08:48.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:08:48.548 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:08:48.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:08:48.614 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:08:48.621 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-09-30T00:08:48.639 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-09-30T00:08:48.639 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-09-30T00:08:48.695 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T00:08:48.695 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.4" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T00:08:48.696 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:08:48.697 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:08:48.697 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:08:48.697 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T00:08:48.697 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T00:08:48.697 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T00:08:48.697 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-09-30T00:08:48.722 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2023-09-30T00:08:48.740 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2023-09-30T00:08:48.740 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2023-09-30T00:08:48.794 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T00:08:48.794 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T00:08:48.794 INFO:teuthology.orchestra.run.smithi146.stdout:ID="rhel" 2023-09-30T00:08:48.794 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="fedora" 2023-09-30T00:08:48.794 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8.4" 2023-09-30T00:08:48.795 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:08:48.795 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T00:08:48.795 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2023-09-30T00:08:48.795 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T00:08:48.795 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T00:08:48.795 INFO:teuthology.orchestra.run.smithi146.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T00:08:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:08:48.796 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:08:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:08:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T00:08:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T00:08:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T00:08:48.797 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2023-09-30T00:08:48.822 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:08:48.831 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:08:48.848 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:08:48.848 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:08:48.851 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:08:48.867 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:08:48.872 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:08:48.872 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:08:48.908 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:08:48.922 INFO:teuthology.orchestra.run.smithi031.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T00:08:48.924 INFO:teuthology.orchestra.run.smithi146.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T00:08:48.924 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:08:48.981 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:08:48.997 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:08:48.998 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:08:48.998 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:08:48.998 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:08:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:08:49.015 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-09-30T00:08:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:08:49.016 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2023-09-30T00:08:49.072 INFO:teuthology.orchestra.run.smithi146.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:08:49.072 INFO:teuthology.task.kernel:Running kernel on smithi146: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:08:49.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2023-09-30T00:08:49.073 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:08:49.073 INFO:teuthology.task.kernel:Running kernel on smithi031: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:08:49.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-09-30T00:08:50.522 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:08:50.523 INFO:teuthology.orchestra.run.smithi146.stdout:Unable to read consumer identity 2023-09-30T00:08:50.572 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:08:50.573 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-09-30T00:08:51.930 INFO:teuthology.orchestra.run.smithi146.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-09-30T00:08:52.005 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 32 kB/s | 3.5 kB 00:00 2023-09-30T00:08:52.543 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 28 MB/s | 16 MB 00:00 2023-09-30T00:08:53.017 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-09-30T00:08:55.844 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-09-30T00:08:56.496 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-09-30T00:08:56.519 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T00:08:56.519 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T00:08:56.519 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T00:08:56.665 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:08:56.666 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:08:56.666 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:08:56.768 DEBUG:teuthology.orchestra.run.smithi146:> echo no | sudo yum reinstall kernel || true 2023-09-30T00:08:57.113 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:08:57.113 INFO:teuthology.orchestra.run.smithi146.stdout:Unable to read consumer identity 2023-09-30T00:08:57.261 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T00:08:57.261 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T00:08:57.261 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T00:08:57.402 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 12:08:55 AM UTC. 2023-09-30T00:08:57.417 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:08:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:08:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:08:57.515 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2023-09-30T00:08:57.552 INFO:teuthology.orchestra.run.smithi146.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:08:57.552 INFO:teuthology.orchestra.run.smithi146.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:57.552 INFO:teuthology.orchestra.run.smithi146.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:57.584 INFO:teuthology.orchestra.run.smithi146.stderr:Error: No packages marked for reinstall. 2023-09-30T00:08:57.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum reinstall -y kernel || true 2023-09-30T00:08:57.881 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:08:57.882 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-09-30T00:08:57.976 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:08:57.976 INFO:teuthology.orchestra.run.smithi146.stdout:Unable to read consumer identity 2023-09-30T00:08:58.190 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 12:08:56 AM UTC. 2023-09-30T00:08:58.269 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:03 ago on Sat 30 Sep 2023 12:08:55 AM UTC. 2023-09-30T00:08:58.317 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:08:58.317 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:58.317 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:58.344 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-09-30T00:08:58.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2023-09-30T00:08:58.397 INFO:teuthology.orchestra.run.smithi146.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:08:58.397 INFO:teuthology.orchestra.run.smithi146.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:58.397 INFO:teuthology.orchestra.run.smithi146.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:58.409 INFO:teuthology.orchestra.run.smithi146.stderr:Error: No packages marked for reinstall. 2023-09-30T00:08:58.460 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:08:58.585 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:58.585 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:58.586 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:58.586 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:08:58.586 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:08:58.586 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T00:08:58.586 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:08:58.587 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:08:58.711 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:58.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:58.712 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2023-09-30T00:08:58.738 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2023-09-30T00:08:58.739 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2023-09-30T00:08:58.739 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.02 2023-09-30T00:08:58.739 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 99.el8_4.1 2023-09-30T00:08:58.739 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2023-09-30T00:08:58.739 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T00:08:58.739 INFO:teuthology.orchestra.run.smithi146.stdout:Group : System Environment/Base 2023-09-30T00:08:58.740 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 9264631 2023-09-30T00:08:58.740 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2023-09-30T00:08:58.740 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T00:08:58.740 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T00:08:58.740 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T00:08:58.740 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T00:08:58.741 INFO:teuthology.orchestra.run.smithi146.stdout:Relocations : (not relocatable) 2023-09-30T00:08:58.741 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : Red Hat, Inc. 2023-09-30T00:08:58.741 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : Red Hat, Inc. 2023-09-30T00:08:58.741 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:08:58.741 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2023-09-30T00:08:58.741 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2023-09-30T00:08:58.742 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:08:58.742 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:08:58.742 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:08:58.742 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:08:58.742 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2023-09-30T00:08:58.743 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:08:58.743 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:08:58.744 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:08:58.744 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:58.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:08:58.761 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:08:58.761 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-09-30T00:08:59.059 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Sat 30 Sep 2023 12:08:56 AM UTC. 2023-09-30T00:08:59.191 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:08:59.191 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:59.191 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:08:59.204 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-09-30T00:08:59.255 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:08:59.385 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:59.386 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:59.386 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:59.386 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:08:59.386 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:08:59.386 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:08:59.386 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:08:59.386 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:08:59.517 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:59.517 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:59.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-09-30T00:08:59.545 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-09-30T00:08:59.545 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-09-30T00:08:59.545 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.02 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 99.el8_4.1 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Group : System Environment/Base 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 9264631 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T00:08:59.546 INFO:teuthology.orchestra.run.smithi031.stdout:Relocations : (not relocatable) 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : Red Hat, Inc. 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : Red Hat, Inc. 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:08:59.547 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:08:59.548 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:08:59.548 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:08:59.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:09:00.130 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2023-09-30T00:09:00.673 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-09-30T00:09:02.329 INFO:teuthology.orchestra.run.smithi146.stderr:done 2023-09-30T00:09:02.331 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2023-09-30T00:09:02.344 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.g6Z274RNDQ 2023-09-30T00:09:02.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.g6Z274RNDQ 2023-09-30T00:09:02.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.g6Z274RNDQ 2023-09-30T00:09:02.566 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.g6Z274RNDQ is 6KB 2023-09-30T00:09:02.615 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.g6Z274RNDQ 2023-09-30T00:09:02.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:09:02.700 INFO:teuthology.orchestra.run.smithi146.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T00:09:02.701 INFO:teuthology.orchestra.run.smithi146.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T00:09:02.701 INFO:teuthology.orchestra.run.smithi146.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T00:09:02.701 INFO:teuthology.orchestra.run.smithi146.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T00:09:02.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:09:02.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2023-09-30T00:09:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-09-30T00:09:02.887 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-09-30T00:09:02.943 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.lF9tKbdyft 2023-09-30T00:09:02.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lF9tKbdyft 2023-09-30T00:09:02.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.lF9tKbdyft 2023-09-30T00:09:03.148 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.lF9tKbdyft is 6KB 2023-09-30T00:09:03.197 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.lF9tKbdyft 2023-09-30T00:09:03.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:09:03.273 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T00:09:03.273 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T00:09:03.273 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T00:09:03.274 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T00:09:03.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:09:03.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-09-30T00:09:32.864 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T00:09:32.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-09-30T00:09:32.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:09:33.435 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:09:33.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-30T00:09:33.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:10:07.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-09-30T00:10:16.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-09-30T00:10:16.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:10:33.454 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:10:34.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-09-30T00:10:42.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-30T00:10:42.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:10:43.871 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-09-30T00:10:44.329 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-30T00:10:44.329 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T00:10:44.330 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-09-30T00:10:44.344 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T00:10:44.344 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:10:44.344 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:10:44.344 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T00:10:45.345 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:10:45.346 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:10:45.429 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-09-30T00:10:45.455 DEBUG:teuthology.parallel:result is None 2023-09-30T00:10:46.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-09-30T00:10:46.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:10:47.120 DEBUG:teuthology.orchestra.run.smithi146:> true 2023-09-30T00:10:47.372 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-09-30T00:10:47.372 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T00:10:47.372 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2023-09-30T00:10:47.387 INFO:teuthology.orchestra.run.smithi146.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T00:10:47.387 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:10:47.387 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:10:47.387 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T00:10:48.389 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T00:10:48.389 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:10:48.513 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2023-09-30T00:10:48.531 DEBUG:teuthology.parallel:result is None 2023-09-30T00:10:48.531 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:10:48.540 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:10:48.540 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:10:48.542 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:10:48.558 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:10:48.564 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:10:48.573 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:10:48.574 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:10:48.601 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:10:48.628 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:10:48.635 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:10:48.635 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:10:48.665 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:10:48.702 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:10:48.710 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:10:48.715 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:10:48.722 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:10:48.723 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:10:48.730 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:10:48.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:10:48.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:10:48.787 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:10:48.796 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:10:48.796 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:10:48.830 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:10:48.844 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:10:48.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:10:48.976 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:10:49.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:10:49.101 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:10:49.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:10:49.168 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:10:49.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:10:49.336 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:10:49.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:10:49.418 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:10:49.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:10:49.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-09-30T00:10:49.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2023-09-30T00:10:49.548 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:10:49.609 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:10:49.915 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:10:49.923 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:10:49.923 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:10:49.962 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:10:49.969 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:10:49.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-09-30T00:10:50.034 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: [ OK ] 2023-09-30T00:10:50.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2023-09-30T00:10:50.186 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: [ OK ] 2023-09-30T00:10:50.187 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:10:50.187 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-09-30T00:10:50.209 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-09-30T00:10:50.210 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2023-09-30T00:10:50.232 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2023-09-30T00:10:50.232 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:10:50.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:10:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:10:50.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:10:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:10:50.301 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:10:50.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:10:50.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:10:50.368 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:10:50.377 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-30T00:10:50.382 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-30T00:10:50.535 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:10:50.552 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:10:50.552 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:19:07.644 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2023-09-30T00:19:07.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-30T00:19:07.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:19:07.718 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-09-30T00:19:07.741 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-30T00:19:07.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-09-30T00:19:07.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:19:07.809 DEBUG:teuthology.orchestra.run.smithi146:> true 2023-09-30T00:19:07.873 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-09-30T00:19:07.873 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:19:07.884 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:19:07.885 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:19:07.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:19:07.887 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:19:07.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:19:07.916 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:19:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:19:07.957 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:19:07.968 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-09-30T00:19:07.979 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-09-30T00:19:07.988 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2023-09-30T00:19:07.993 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:19:07.998 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2023-09-30T00:19:08.008 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:19:08.013 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:19:08.026 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:19:08.077 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-09-30T00:19:08.079 INFO:teuthology.orchestra.run.smithi031.stdout:210 Number of sources = 4 2023-09-30T00:19:08.079 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:19:08.080 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-09-30T00:19:08.080 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.080 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.080 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.080 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.106 INFO:teuthology.orchestra.run.smithi146.stderr:bash: ntpq: command not found 2023-09-30T00:19:08.108 INFO:teuthology.orchestra.run.smithi146.stdout:210 Number of sources = 4 2023-09-30T00:19:08.109 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:19:08.109 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2023-09-30T00:19:08.109 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.109 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.109 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.109 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:19:08.110 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:19:08.119 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:19:08.119 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:19:08.119 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:19:08.119 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:19:08.125 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:19:08.125 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:19:08.125 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:19:08.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:19:08.128 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:19:08.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:19:08.364 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:19:08.364 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:19:08.370 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:19:08.370 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:19:08.473 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:19:08.473 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:19:08.474 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:19:08.500 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:19:08.501 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:19:08.527 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:19:08.527 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:19:08.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:19:08.553 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:19:08.553 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:19:08.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:19:08.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:19:08.643 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:19:08.692 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:19:08.708 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-09-30T00:19:08.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-09-30T00:19:08.755 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2023-09-30T00:19:08.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2023-09-30T00:19:09.118 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:19:09.147 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:19:09.426 INFO:teuthology.orchestra.run.smithi031.stdout:44 files removed 2023-09-30T00:19:09.442 INFO:teuthology.orchestra.run.smithi146.stdout:44 files removed 2023-09-30T00:19:09.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2023-09-30T00:19:09.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2023-09-30T00:19:09.848 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:19:09.850 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:19:10.534 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 356 kB/s | 78 kB 00:00 2023-09-30T00:19:10.656 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 250 kB/s | 78 kB 00:00 2023-09-30T00:19:10.815 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 38 kB/s | 9.2 kB 00:00 2023-09-30T00:19:10.885 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 49 kB/s | 9.2 kB 00:00 2023-09-30T00:19:11.151 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 4.8 kB/s | 1.5 kB 00:00 2023-09-30T00:19:11.219 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 4.8 kB/s | 1.5 kB 00:00 2023-09-30T00:19:11.261 INFO:teuthology.orchestra.run.smithi146.stdout:Copr repo for python3-asyncssh owned by ceph 43 kB/s | 3.5 kB 00:00 2023-09-30T00:19:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-09-30T00:19:11.690 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 40 MB/s | 16 MB 00:00 2023-09-30T00:19:13.262 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 16 MB 00:01 2023-09-30T00:19:14.951 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 1.7 MB/s | 24 kB 00:00 2023-09-30T00:19:15.335 INFO:teuthology.orchestra.run.smithi146.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 126 MB/s | 38 MB 00:00 2023-09-30T00:19:16.760 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-09-30T00:19:17.199 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 MB/s | 33 MB 00:00 2023-09-30T00:19:20.859 INFO:teuthology.orchestra.run.smithi146.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-09-30T00:19:22.823 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 109 MB/s | 38 MB 00:00 2023-09-30T00:19:29.410 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:19:29.412 INFO:teuthology.orchestra.run.smithi146.stdout:==================================================================================================== 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout:==================================================================================================== 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:19:29.413 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:19:29.414 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout:==================================================================================================== 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout:Install 21 Packages 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2023-09-30T00:19:29.415 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:19:29.416 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 63 M 2023-09-30T00:19:29.416 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:19:30.603 INFO:teuthology.orchestra.run.smithi146.stdout:(1/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 17 MB/s | 17 MB 00:01 2023-09-30T00:19:30.624 INFO:teuthology.orchestra.run.smithi146.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 24 kB 00:00 2023-09-30T00:19:30.664 INFO:teuthology.orchestra.run.smithi146.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 20 MB/s | 796 kB 00:00 2023-09-30T00:19:30.699 INFO:teuthology.orchestra.run.smithi146.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 14 MB/s | 506 kB 00:00 2023-09-30T00:19:30.993 INFO:teuthology.orchestra.run.smithi146.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 19 MB/s | 5.6 MB 00:00 2023-09-30T00:19:31.014 INFO:teuthology.orchestra.run.smithi146.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 2.1 MB/s | 45 kB 00:00 2023-09-30T00:19:31.036 INFO:teuthology.orchestra.run.smithi146.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 5.3 MB/s | 115 kB 00:00 2023-09-30T00:19:31.058 INFO:teuthology.orchestra.run.smithi146.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 8.6 MB/s | 190 kB 00:00 2023-09-30T00:19:31.078 INFO:teuthology.orchestra.run.smithi146.stdout:(9/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 3.5 MB/s | 5.2 MB 00:01 2023-09-30T00:19:31.082 INFO:teuthology.orchestra.run.smithi146.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 15 MB/s | 370 kB 00:00 2023-09-30T00:19:31.119 INFO:teuthology.orchestra.run.smithi146.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.8 MB/s | 107 kB 00:00 2023-09-30T00:19:31.131 INFO:teuthology.orchestra.run.smithi146.stdout:(12/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 6.6 MB/s | 358 kB 00:00 2023-09-30T00:19:31.621 INFO:teuthology.orchestra.run.smithi146.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 120 kB/s | 59 kB 00:00 2023-09-30T00:19:31.782 INFO:teuthology.orchestra.run.smithi146.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 468 kB/s | 75 kB 00:00 2023-09-30T00:19:31.844 INFO:teuthology.orchestra.run.smithi146.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 423 kB/s | 306 kB 00:00 2023-09-30T00:19:31.877 INFO:teuthology.orchestra.run.smithi146.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-09-30T00:19:31.908 INFO:teuthology.orchestra.run.smithi146.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2023-09-30T00:19:31.942 INFO:teuthology.orchestra.run.smithi146.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.5 MB/s | 259 kB 00:00 2023-09-30T00:19:31.977 INFO:teuthology.orchestra.run.smithi146.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2023-09-30T00:19:32.110 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:19:32.113 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:19:32.114 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-09-30T00:19:32.115 INFO:teuthology.orchestra.run.smithi031.stdout:Install 21 Packages 2023-09-30T00:19:32.116 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2023-09-30T00:19:32.116 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:19:32.116 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 63 M 2023-09-30T00:19:32.116 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:19:32.208 INFO:teuthology.orchestra.run.smithi146.stdout:(20/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 9.0 MB/s | 23 MB 00:02 2023-09-30T00:19:32.405 INFO:teuthology.orchestra.run.smithi146.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 18 MB/s | 3.6 MB 00:00 2023-09-30T00:19:32.558 INFO:teuthology.orchestra.run.smithi146.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 6.0 MB/s | 3.5 MB 00:00 2023-09-30T00:19:33.322 INFO:teuthology.orchestra.run.smithi031.stdout:(1/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 15 MB/s | 17 MB 00:01 2023-09-30T00:19:33.344 INFO:teuthology.orchestra.run.smithi031.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 24 kB 00:00 2023-09-30T00:19:33.386 INFO:teuthology.orchestra.run.smithi031.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 19 MB/s | 796 kB 00:00 2023-09-30T00:19:33.422 INFO:teuthology.orchestra.run.smithi031.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 14 MB/s | 506 kB 00:00 2023-09-30T00:19:33.533 INFO:teuthology.orchestra.run.smithi031.stdout:(5/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 3.8 MB/s | 5.2 MB 00:01 2023-09-30T00:19:33.605 INFO:teuthology.orchestra.run.smithi031.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 618 kB/s | 45 kB 00:00 2023-09-30T00:19:33.686 INFO:teuthology.orchestra.run.smithi031.stdout:(7/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 21 MB/s | 5.6 MB 00:00 2023-09-30T00:19:33.687 INFO:teuthology.orchestra.run.smithi031.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 1.4 MB/s | 115 kB 00:00 2023-09-30T00:19:33.708 INFO:teuthology.orchestra.run.smithi031.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 8.2 MB/s | 190 kB 00:00 2023-09-30T00:19:33.742 INFO:teuthology.orchestra.run.smithi031.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 11 MB/s | 358 kB 00:00 2023-09-30T00:19:33.762 INFO:teuthology.orchestra.run.smithi031.stdout:(11/23): python3-rados-17.2.6-1165.g93dce5e1.el 4.8 MB/s | 370 kB 00:00 2023-09-30T00:19:33.763 INFO:teuthology.orchestra.run.smithi031.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 4.9 MB/s | 107 kB 00:00 2023-09-30T00:19:34.025 INFO:teuthology.orchestra.run.smithi146.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 792 kB/s | 1.7 MB 00:02 2023-09-30T00:19:34.030 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:19:34.031 INFO:teuthology.orchestra.run.smithi146.stdout:Total 14 MB/s | 63 MB 00:04 2023-09-30T00:19:34.035 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:19:34.080 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:19:34.080 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:19:34.121 INFO:teuthology.orchestra.run.smithi031.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 165 kB/s | 59 kB 00:00 2023-09-30T00:19:34.273 INFO:teuthology.orchestra.run.smithi031.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 495 kB/s | 75 kB 00:00 2023-09-30T00:19:34.282 INFO:teuthology.orchestra.run.smithi031.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 589 kB/s | 306 kB 00:00 2023-09-30T00:19:34.323 INFO:teuthology.orchestra.run.smithi031.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-09-30T00:19:34.343 INFO:teuthology.orchestra.run.smithi031.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 13 MB/s | 259 kB 00:00 2023-09-30T00:19:34.364 INFO:teuthology.orchestra.run.smithi031.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 17 MB/s | 353 kB 00:00 2023-09-30T00:19:34.378 INFO:teuthology.orchestra.run.smithi031.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.3 MB/s | 47 kB 00:00 2023-09-30T00:19:34.646 INFO:teuthology.orchestra.run.smithi031.stdout:(20/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 9.5 MB/s | 23 MB 00:02 2023-09-30T00:19:34.725 INFO:teuthology.orchestra.run.smithi031.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.8 MB/s | 1.7 MB 00:00 2023-09-30T00:19:34.815 INFO:teuthology.orchestra.run.smithi031.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 21 MB/s | 3.6 MB 00:00 2023-09-30T00:19:34.815 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:19:34.819 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:19:35.109 INFO:teuthology.orchestra.run.smithi031.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 4.7 MB/s | 3.5 MB 00:00 2023-09-30T00:19:35.111 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:19:35.112 INFO:teuthology.orchestra.run.smithi031.stdout:Total 21 MB/s | 63 MB 00:02 2023-09-30T00:19:35.116 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:19:35.164 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:19:35.164 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:19:35.884 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:19:35.888 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:19:36.034 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:19:36.261 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:19:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:19:36.869 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:19:36.971 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:19:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:19:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:19:37.259 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:19:37.460 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T00:19:37.671 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:19:37.693 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:19:37.959 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:19:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:19:38.118 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:19:38.257 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:19:38.369 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:19:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:19:38.540 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T00:19:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-30T00:19:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-30T00:19:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:19:39.073 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-09-30T00:19:39.272 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:19:39.592 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:19:39.662 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:19:39.756 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T00:19:39.840 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-09-30T00:19:39.918 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:19:39.974 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:19:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:19:40.137 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:19:40.289 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:19:40.338 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:19:40.579 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:19:40.822 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:19:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:19:41.033 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T00:19:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:19:41.191 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:19:41.263 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:19:41.292 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:19:41.437 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:19:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T00:19:41.629 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:19:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:19:41.863 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:19:41.977 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:19:42.091 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:19:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:19:42.282 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:19:42.479 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:19:42.714 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T00:19:42.992 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:19:43.167 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:19:47.037 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:19:47.309 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:19:47.929 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:19:48.233 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:19:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:19:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:19:48.544 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:19:48.774 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:19:49.409 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:19:49.735 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:19:49.824 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:19:49.982 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /dev 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /run 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2023-09-30T00:20:06.325 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2023-09-30T00:20:06.326 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2023-09-30T00:20:06.326 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2023-09-30T00:20:06.326 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /dev 2023-09-30T00:20:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:20:08.099 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:20:08.100 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:20:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:20:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /run 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-09-30T00:20:08.426 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-09-30T00:20:08.427 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-09-30T00:20:08.427 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-09-30T00:20:08.427 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-09-30T00:20:08.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:20:09.143 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-09-30T00:20:09.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-09-30T00:20:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-09-30T00:20:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-09-30T00:20:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:20:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:20:09.145 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:20:10.264 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:20:10.344 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:20:10.344 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:20:10.497 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:20:10.497 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:50 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:10.539 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:20:10.540 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:20:11.371 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:20:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:20:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:20:12.453 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:20:12.561 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.636 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.637 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:20:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:20:12.638 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:12.638 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:20:12.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2023-09-30T00:20:12.902 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:20:12.902 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:50 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:13.062 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:13.907 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:53 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:14.855 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:20:14.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.980 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.981 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.982 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:14.982 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:20:14.982 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:14.982 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:20:15.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2023-09-30T00:20:15.510 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:15.741 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:20:15.742 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2023-09-30T00:20:15.743 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:15.744 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 53 M 2023-09-30T00:20:15.744 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 233 M 2023-09-30T00:20:15.744 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:20:15.748 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 17 MB/s | 70 kB 00:00 2023-09-30T00:20:15.803 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 2.9 MB/s | 161 kB 00:00 2023-09-30T00:20:15.834 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.8 MB/s | 187 kB 00:00 2023-09-30T00:20:15.867 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 9.1 MB/s | 302 kB 00:00 2023-09-30T00:20:15.904 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 175 kB 00:00 2023-09-30T00:20:16.407 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:54 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:18.078 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 22 MB/s | 52 MB 00:02 2023-09-30T00:20:18.081 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:20:18.081 INFO:teuthology.orchestra.run.smithi146.stdout:Total 22 MB/s | 53 MB 00:02 2023-09-30T00:20:18.085 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:20:18.108 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:20:18.108 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:20:18.347 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:20:18.348 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-09-30T00:20:18.348 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:18.349 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:20:18.350 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-09-30T00:20:18.350 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-09-30T00:20:18.350 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:18.350 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 53 M 2023-09-30T00:20:18.350 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 233 M 2023-09-30T00:20:18.351 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:20:18.358 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 17 MB/s | 70 kB 00:00 2023-09-30T00:20:18.395 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.0 MB/s | 161 kB 00:00 2023-09-30T00:20:18.411 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-09-30T00:20:18.431 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 16 MB/s | 302 kB 00:00 2023-09-30T00:20:18.478 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:20:18.482 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:20:18.627 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 632 kB/s | 175 kB 00:00 2023-09-30T00:20:19.011 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:20:19.241 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T00:20:19.445 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:20:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:20:19.702 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T00:20:19.903 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T00:20:20.062 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 22 MB/s | 52 MB 00:02 2023-09-30T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:Total 22 MB/s | 53 MB 00:02 2023-09-30T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:20:21.145 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:20:21.152 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:20:21.720 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:20:21.940 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T00:20:22.168 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:20:22.251 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:20:22.453 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T00:20:22.635 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T00:20:22.818 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:20:25.086 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:20:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:20:25.671 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:20:25.671 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:20:25.672 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:20:25.672 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T00:20:25.672 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T00:20:25.672 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T00:20:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T00:20:26.664 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:06 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:28.135 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:20:28.433 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:20:28.511 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.5-12.el8.x86_64 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:20:28.610 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T00:20:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:20:28.611 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:28.611 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:20:28.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2023-09-30T00:20:28.735 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:20:28.736 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:20:28.736 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:20:28.736 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T00:20:28.736 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T00:20:28.736 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T00:20:29.073 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:29.866 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:09 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:29.902 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T00:20:29.902 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:07 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:31.759 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:20:31.762 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================================= 2023-09-30T00:20:31.762 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:20:31.762 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================================= 2023-09-30T00:20:31.762 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:20:31.763 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T00:20:31.764 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:20:31.765 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================================= 2023-09-30T00:20:31.766 INFO:teuthology.orchestra.run.smithi146.stdout:Install 28 Packages 2023-09-30T00:20:31.766 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:31.766 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 27 M 2023-09-30T00:20:31.766 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 106 M 2023-09-30T00:20:31.766 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:20:31.871 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:20:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:31.978 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:20:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.5-12.el8.x86_64 2023-09-30T00:20:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:20:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T00:20:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:20:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:31.979 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:20:32.056 INFO:teuthology.orchestra.run.smithi146.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-09-30T00:20:32.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2023-09-30T00:20:32.166 INFO:teuthology.orchestra.run.smithi146.stdout:(2/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 8.9 MB/s | 2.3 MB 00:00 2023-09-30T00:20:32.480 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:32.686 INFO:teuthology.orchestra.run.smithi146.stdout:(3/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 2.0 MB/s | 1.6 MB 00:00 2023-09-30T00:20:32.964 INFO:teuthology.orchestra.run.smithi146.stdout:(4/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 21 MB/s | 17 MB 00:00 2023-09-30T00:20:32.966 INFO:teuthology.orchestra.run.smithi146.stdout:(5/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 852 kB/s | 238 kB 00:00 2023-09-30T00:20:33.042 INFO:teuthology.orchestra.run.smithi146.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 561 kB/s | 44 kB 00:00 2023-09-30T00:20:33.082 INFO:teuthology.orchestra.run.smithi146.stdout:(7/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.6 MB/s | 185 kB 00:00 2023-09-30T00:20:33.102 INFO:teuthology.orchestra.run.smithi146.stdout:(8/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.8 MB/s | 173 kB 00:00 2023-09-30T00:20:33.122 INFO:teuthology.orchestra.run.smithi146.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 560 kB/s | 11 kB 00:00 2023-09-30T00:20:33.133 INFO:teuthology.orchestra.run.smithi146.stdout:(10/28): python3-cherrypy-18.4.0-1.el8.noarch.r 7.3 MB/s | 384 kB 00:00 2023-09-30T00:20:33.143 INFO:teuthology.orchestra.run.smithi146.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 878 kB/s | 18 kB 00:00 2023-09-30T00:20:33.153 INFO:teuthology.orchestra.run.smithi146.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 2.4 MB/s | 49 kB 00:00 2023-09-30T00:20:33.165 INFO:teuthology.orchestra.run.smithi146.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 2.6 MB/s | 59 kB 00:00 2023-09-30T00:20:33.182 INFO:teuthology.orchestra.run.smithi146.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 9.7 MB/s | 283 kB 00:00 2023-09-30T00:20:33.184 INFO:teuthology.orchestra.run.smithi146.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 789 kB/s | 16 kB 00:00 2023-09-30T00:20:33.202 INFO:teuthology.orchestra.run.smithi146.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 972 kB/s | 19 kB 00:00 2023-09-30T00:20:33.204 INFO:teuthology.orchestra.run.smithi146.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2023-09-30T00:20:33.221 INFO:teuthology.orchestra.run.smithi146.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 1.5 MB/s | 29 kB 00:00 2023-09-30T00:20:33.224 INFO:teuthology.orchestra.run.smithi146.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2023-09-30T00:20:33.249 INFO:teuthology.orchestra.run.smithi146.stdout:(20/28): python3-waitress-1.4.3-1.el8.noarch.rp 9.2 MB/s | 255 kB 00:00 2023-09-30T00:20:33.260 INFO:teuthology.orchestra.run.smithi146.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.9 MB/s | 251 kB 00:00 2023-09-30T00:20:33.270 INFO:teuthology.orchestra.run.smithi146.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 4.0 MB/s | 85 kB 00:00 2023-09-30T00:20:33.280 INFO:teuthology.orchestra.run.smithi146.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 1.1 MB/s | 23 kB 00:00 2023-09-30T00:20:33.325 INFO:teuthology.orchestra.run.smithi146.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 8.1 MB/s | 457 kB 00:00 2023-09-30T00:20:33.337 INFO:teuthology.orchestra.run.smithi146.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 689 kB/s | 39 kB 00:00 2023-09-30T00:20:33.359 INFO:teuthology.orchestra.run.smithi146.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 4.5 MB/s | 157 kB 00:00 2023-09-30T00:20:33.365 INFO:teuthology.orchestra.run.smithi146.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 1.9 MB/s | 54 kB 00:00 2023-09-30T00:20:33.404 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:11 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:33.618 INFO:teuthology.orchestra.run.smithi146.stdout:(28/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 2.5 MB/s | 3.9 MB 00:01 2023-09-30T00:20:33.620 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:20:33.621 INFO:teuthology.orchestra.run.smithi146.stdout:Total 15 MB/s | 27 MB 00:01 2023-09-30T00:20:33.625 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:20:33.661 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:20:33.661 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:20:34.056 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:20:34.061 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:20:35.232 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:20:35.311 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:20:35.314 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-09-30T00:20:35.314 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:20:35.314 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-09-30T00:20:35.314 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:20:35.314 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:20:35.314 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:20:35.315 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:20:35.316 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout:Install 28 Packages 2023-09-30T00:20:35.317 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:35.318 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 27 M 2023-09-30T00:20:35.318 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 106 M 2023-09-30T00:20:35.318 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:20:35.475 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T00:20:35.735 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T00:20:35.778 INFO:teuthology.orchestra.run.smithi031.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 21 kB/s | 6.4 kB 00:00 2023-09-30T00:20:35.953 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T00:20:36.208 INFO:teuthology.orchestra.run.smithi031.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 1.6 MB 00:00 2023-09-30T00:20:36.233 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T00:20:36.333 INFO:teuthology.orchestra.run.smithi031.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 2.7 MB/s | 2.3 MB 00:00 2023-09-30T00:20:36.372 INFO:teuthology.orchestra.run.smithi031.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 6.0 MB/s | 238 kB 00:00 2023-09-30T00:20:36.544 INFO:teuthology.orchestra.run.smithi031.stdout:(5/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 5.1 MB/s | 3.9 MB 00:00 2023-09-30T00:20:36.556 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T00:20:36.614 INFO:teuthology.orchestra.run.smithi031.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 181 kB/s | 44 kB 00:00 2023-09-30T00:20:36.690 INFO:teuthology.orchestra.run.smithi031.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.2 MB/s | 173 kB 00:00 2023-09-30T00:20:36.767 INFO:teuthology.orchestra.run.smithi031.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 4.9 MB/s | 384 kB 00:00 2023-09-30T00:20:36.779 INFO:teuthology.orchestra.run.smithi031.stdout:(9/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 790 kB/s | 185 kB 00:00 2023-09-30T00:20:36.805 INFO:teuthology.orchestra.run.smithi031.stdout:(10/28): python3-jaraco-6.2-6.el8.noarch.rpm 297 kB/s | 11 kB 00:00 2023-09-30T00:20:36.817 INFO:teuthology.orchestra.run.smithi031.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 470 kB/s | 18 kB 00:00 2023-09-30T00:20:36.843 INFO:teuthology.orchestra.run.smithi031.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 1.3 MB/s | 49 kB 00:00 2023-09-30T00:20:36.854 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T00:20:36.855 INFO:teuthology.orchestra.run.smithi031.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 1.5 MB/s | 59 kB 00:00 2023-09-30T00:20:36.883 INFO:teuthology.orchestra.run.smithi031.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 6.8 MB/s | 283 kB 00:00 2023-09-30T00:20:36.892 INFO:teuthology.orchestra.run.smithi031.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 415 kB/s | 16 kB 00:00 2023-09-30T00:20:36.921 INFO:teuthology.orchestra.run.smithi031.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 498 kB/s | 19 kB 00:00 2023-09-30T00:20:36.930 INFO:teuthology.orchestra.run.smithi031.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 634 kB/s | 24 kB 00:00 2023-09-30T00:20:36.959 INFO:teuthology.orchestra.run.smithi031.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 771 kB/s | 29 kB 00:00 2023-09-30T00:20:36.968 INFO:teuthology.orchestra.run.smithi031.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 729 kB/s | 27 kB 00:00 2023-09-30T00:20:37.001 INFO:teuthology.orchestra.run.smithi031.stdout:(20/28): python3-waitress-1.4.3-1.el8.noarch.rp 6.1 MB/s | 255 kB 00:00 2023-09-30T00:20:37.039 INFO:teuthology.orchestra.run.smithi031.stdout:(21/28): python3-webtest-2.0.33-1.el8.noarch.rp 2.2 MB/s | 85 kB 00:00 2023-09-30T00:20:37.044 INFO:teuthology.orchestra.run.smithi031.stdout:(22/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2023-09-30T00:20:37.055 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T00:20:37.077 INFO:teuthology.orchestra.run.smithi031.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 611 kB/s | 23 kB 00:00 2023-09-30T00:20:37.085 INFO:teuthology.orchestra.run.smithi031.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 11 MB/s | 457 kB 00:00 2023-09-30T00:20:37.110 INFO:teuthology.orchestra.run.smithi031.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-09-30T00:20:37.112 INFO:teuthology.orchestra.run.smithi031.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 8.1 MB/s | 157 kB 00:00 2023-09-30T00:20:37.125 INFO:teuthology.orchestra.run.smithi031.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.6 MB/s | 54 kB 00:00 2023-09-30T00:20:37.259 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T00:20:37.433 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T00:20:37.446 INFO:teuthology.orchestra.run.smithi031.stdout:(28/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 14 MB/s | 17 MB 00:01 2023-09-30T00:20:37.451 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:20:37.451 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 27 MB 00:02 2023-09-30T00:20:37.456 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:20:37.496 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:20:37.496 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:20:37.606 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T00:20:37.813 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T00:20:37.886 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:20:37.891 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:20:38.021 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T00:20:38.141 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T00:20:38.359 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T00:20:38.580 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T00:20:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T00:20:39.033 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T00:20:39.101 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:20:39.274 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T00:20:39.363 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T00:20:39.477 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T00:20:39.590 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T00:20:39.768 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T00:20:39.797 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T00:20:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T00:20:40.057 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T00:20:40.339 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T00:20:40.358 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T00:20:40.600 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T00:20:40.749 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T00:20:40.853 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T00:20:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:20:41.066 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T00:20:41.240 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T00:20:41.370 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T00:20:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T00:20:41.820 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T00:20:42.006 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T00:20:42.205 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T00:20:42.420 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T00:20:42.465 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:20:42.608 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:20:42.613 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T00:20:42.838 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T00:20:43.004 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:20:43.065 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T00:20:43.159 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:20:43.387 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:20:43.567 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:20:43.585 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:20:43.691 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:20:43.926 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T00:20:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:20:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T00:20:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T00:20:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T00:20:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T00:20:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T00:20:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T00:20:44.529 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T00:20:44.830 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T00:20:45.184 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T00:20:45.342 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:20:45.654 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T00:20:45.655 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:25 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:47.024 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:20:47.208 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:20:47.560 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:20:47.631 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:47.659 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:20:47.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:20:47.661 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:20:47.662 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:20:47.662 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:47.662 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:20:47.840 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:20:47.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2023-09-30T00:20:48.103 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:20:48.213 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:48.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:20:48.292 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:20:48.406 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T00:20:48.856 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T00:20:48.857 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T00:20:48.858 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T00:20:48.994 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:28 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:50.343 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:20:50.362 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T00:20:50.362 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:28 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:50.824 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:20:50.825 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:20:50.825 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:20:50.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2023-09-30T00:20:51.292 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:52.076 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:52.390 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:20:52.497 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:52.497 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:20:52.497 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:52.497 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:20:52.498 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:20:52.499 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:52.500 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:20:52.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2023-09-30T00:20:53.015 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:53.897 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:20:53.898 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:20:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:20:53.898 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 80 k 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 363 k 2023-09-30T00:20:53.899 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:20:53.988 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:54.030 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 611 kB/s | 80 kB 00:00 2023-09-30T00:20:54.031 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:20:54.031 INFO:teuthology.orchestra.run.smithi146.stdout:Total 604 kB/s | 80 kB 00:00 2023-09-30T00:20:54.032 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:20:54.039 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:20:54.039 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:20:54.041 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:20:54.046 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:20:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:20:55.460 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:20:55.973 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:20:55.975 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:20:55.975 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:20:56.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2023-09-30T00:20:56.461 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:20:57.036 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:20:57.177 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:20:57.386 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:20:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:20:58.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:20:58.361 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:20:59.375 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:59.376 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:20:59.377 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:20:59.377 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:20:59.377 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:20:59.377 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 80 k 2023-09-30T00:20:59.377 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 363 k 2023-09-30T00:20:59.377 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:20:59.601 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 357 kB/s | 80 kB 00:00 2023-09-30T00:20:59.602 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:20:59.602 INFO:teuthology.orchestra.run.smithi031.stdout:Total 354 kB/s | 80 kB 00:00 2023-09-30T00:20:59.603 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:20:59.612 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:20:59.612 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:20:59.614 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:20:59.619 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:21:00.080 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:21:00.619 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:21:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:00.716 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:21:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:00.716 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:00.716 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:00.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:21:01.224 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:02.808 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:21:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:21:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:21:04.733 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:21:04.734 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:21:04.735 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:04.735 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 161 k 2023-09-30T00:21:04.735 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 490 k 2023-09-30T00:21:04.735 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:21:04.993 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 625 kB/s | 161 kB 00:00 2023-09-30T00:21:04.993 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:04.994 INFO:teuthology.orchestra.run.smithi146.stdout:Total 622 kB/s | 161 kB 00:00 2023-09-30T00:21:04.994 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:21:05.007 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:21:05.007 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:21:05.058 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:21:05.063 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:21:05.183 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:21:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:21:05.194 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:21:05.761 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:21:06.723 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:21:06.723 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:07.265 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:21:07.371 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:07.371 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:21:07.371 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:07.371 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:07.371 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:21:07.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:21:07.907 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:08.672 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:21:08.768 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:08.768 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:21:08.768 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:21:08.768 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:08.768 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:08.881 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:08.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2023-09-30T00:21:09.285 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:10.058 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:10.880 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:21:10.880 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:10.881 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 161 k 2023-09-30T00:21:10.882 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 490 k 2023-09-30T00:21:10.882 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:21:11.033 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 1.0 MB/s | 161 kB 00:00 2023-09-30T00:21:11.034 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:11.035 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.0 MB/s | 161 kB 00:00 2023-09-30T00:21:11.035 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:21:11.049 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:21:11.049 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:21:11.106 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:21:11.111 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:21:11.234 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:21:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:21:11.523 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:21:11.818 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:21:11.994 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:21:11.995 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:21:11.995 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:12.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:21:12.465 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:21:12.944 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:13.215 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:53 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:15.021 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:21:15.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:15.128 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:21:15.128 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:21:15.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:15.129 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:21:15.153 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:21:15.154 INFO:teuthology.orchestra.run.smithi146.stdout:=================================================================================================== 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout:=================================================================================================== 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:21:15.155 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout:=================================================================================================== 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.0 M 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 8.0 M 2023-09-30T00:21:15.156 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:21:15.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2023-09-30T00:21:15.465 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 117 kB/s | 14 kB 00:00 2023-09-30T00:21:15.465 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 185 kB/s | 23 kB 00:00 2023-09-30T00:21:15.575 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 305 kB/s | 34 kB 00:00 2023-09-30T00:21:15.600 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2023-09-30T00:21:15.659 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.3 MB/s | 1.7 MB 00:00 2023-09-30T00:21:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:15.737 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 264 kB/s | 43 kB 00:00 2023-09-30T00:21:15.740 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:15.740 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2023-09-30T00:21:15.742 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:21:15.750 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:21:15.751 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:21:15.933 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:21:15.938 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:21:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:21:16.570 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T00:21:16.588 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:16.737 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T00:21:16.906 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T00:21:17.059 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:21:17.466 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:21:17.752 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:21:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:21:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:21:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:21:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:21:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T00:21:18.040 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T00:21:18.706 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:21:18.978 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T00:21:18.978 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:58 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:19.226 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:21:19.227 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:21:19.227 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:21:19.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:21:19.706 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:58 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:20.884 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:21:20.980 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:20.980 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:21:20.980 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:20.981 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:21.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:21:21.479 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:22.230 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:02 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:22.583 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:21:22.584 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-09-30T00:21:22.584 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:21:22.584 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-09-30T00:21:22.584 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:21:22.584 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-09-30T00:21:22.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:22.586 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2023-09-30T00:21:22.586 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.0 M 2023-09-30T00:21:22.586 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:21:23.042 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 208 kB/s | 23 kB 00:00 2023-09-30T00:21:23.122 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 76 kB/s | 14 kB 00:00 2023-09-30T00:21:23.156 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 295 kB/s | 34 kB 00:00 2023-09-30T00:21:23.196 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-09-30T00:21:23.304 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-09-30T00:21:23.337 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 4.2 MB/s | 1.7 MB 00:00 2023-09-30T00:21:23.339 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:23.339 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-09-30T00:21:23.341 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:21:23.350 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:21:23.351 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:21:23.440 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:21:23.444 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:21:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:21:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T00:21:24.113 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:21:24.114 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================================================== 2023-09-30T00:21:24.114 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================================================== 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T00:21:24.115 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:24.116 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:21:24.116 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================================================== 2023-09-30T00:21:24.116 INFO:teuthology.orchestra.run.smithi146.stdout:Install 8 Packages 2023-09-30T00:21:24.116 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:24.116 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 36 M 2023-09-30T00:21:24.117 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 199 M 2023-09-30T00:21:24.117 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:21:24.167 INFO:teuthology.orchestra.run.smithi146.stdout:(1/8): libquadmath-8.4.1-1.el8.x86_64.rpm 3.3 MB/s | 169 kB 00:00 2023-09-30T00:21:24.171 INFO:teuthology.orchestra.run.smithi146.stdout:(2/8): libgfortran-8.4.1-1.el8.x86_64.rpm 11 MB/s | 639 kB 00:00 2023-09-30T00:21:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T00:21:24.207 INFO:teuthology.orchestra.run.smithi146.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 6.2 MB/s | 225 kB 00:00 2023-09-30T00:21:24.285 INFO:teuthology.orchestra.run.smithi146.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 48 MB/s | 3.7 MB 00:00 2023-09-30T00:21:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T00:21:24.478 INFO:teuthology.orchestra.run.smithi146.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 47 MB/s | 15 MB 00:00 2023-09-30T00:21:24.505 INFO:teuthology.orchestra.run.smithi146.stdout:(6/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-09-30T00:21:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:21:24.549 INFO:teuthology.orchestra.run.smithi146.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 65 MB/s | 4.6 MB 00:00 2023-09-30T00:21:24.832 INFO:teuthology.orchestra.run.smithi146.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 10 MB/s | 7.4 MB 00:00 2023-09-30T00:21:24.833 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:24.834 INFO:teuthology.orchestra.run.smithi146.stdout:Total 50 MB/s | 36 MB 00:00 2023-09-30T00:21:24.843 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:21:24.862 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:21:24.862 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:21:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:21:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:21:25.272 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:21:25.276 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:21:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:21:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:21:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:21:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:21:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T00:21:25.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T00:21:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:21:26.177 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:21:26.315 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:21:26.478 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T00:21:26.478 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:04 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:26.509 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:21:27.063 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:21:27.226 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:21:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:21:28.026 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:21:28.449 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:21:28.557 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:21:28.558 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:21:28.558 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:28.558 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:21:28.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:21:28.775 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:21:29.055 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:29.128 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:21:29.359 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:21:29.915 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:07 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:31.437 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:21:31.925 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:21:31.926 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================================================== 2023-09-30T00:21:31.926 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:21:31.926 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================================================== 2023-09-30T00:21:31.926 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:31.927 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:21:31.928 INFO:teuthology.orchestra.run.smithi031.stdout:====================================================================================================================== 2023-09-30T00:21:31.928 INFO:teuthology.orchestra.run.smithi031.stdout:Install 8 Packages 2023-09-30T00:21:31.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:31.928 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 36 M 2023-09-30T00:21:31.928 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 199 M 2023-09-30T00:21:31.928 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:21:31.979 INFO:teuthology.orchestra.run.smithi031.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 4.4 MB/s | 225 kB 00:00 2023-09-30T00:21:32.052 INFO:teuthology.orchestra.run.smithi031.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 52 MB/s | 3.7 MB 00:00 2023-09-30T00:21:32.141 INFO:teuthology.orchestra.run.smithi031.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 53 MB/s | 4.7 MB 00:00 2023-09-30T00:21:32.292 INFO:teuthology.orchestra.run.smithi031.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 40 MB/s | 15 MB 00:00 2023-09-30T00:21:32.311 INFO:teuthology.orchestra.run.smithi031.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 8.8 MB/s | 169 kB 00:00 2023-09-30T00:21:32.338 INFO:teuthology.orchestra.run.smithi031.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 23 MB/s | 639 kB 00:00 2023-09-30T00:21:32.354 INFO:teuthology.orchestra.run.smithi031.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 22 MB/s | 4.6 MB 00:00 2023-09-30T00:21:32.480 INFO:teuthology.orchestra.run.smithi031.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 13 MB/s | 7.4 MB 00:00 2023-09-30T00:21:32.482 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:32.482 INFO:teuthology.orchestra.run.smithi031.stdout:Total 65 MB/s | 36 MB 00:00 2023-09-30T00:21:32.492 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:21:32.513 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:21:32.513 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:21:32.731 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T00:21:32.925 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:21:32.960 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:21:32.965 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:21:33.234 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:21:33.234 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:21:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:21:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-09-30T00:21:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-09-30T00:21:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:21:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:21:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-09-30T00:21:33.605 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:21:33.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:21:34.208 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-09-30T00:21:34.209 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:14 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:34.333 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:21:34.489 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:21:35.099 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:21:35.256 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:21:35.819 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:21:36.022 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:21:36.068 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T00:21:36.168 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T00:21:36.169 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T00:21:36.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:36.169 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:36.320 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:21:36.684 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:21:37.103 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:21:37.348 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:21:37.442 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:17 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:39.348 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:21:39.350 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================================== 2023-09-30T00:21:39.350 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:21:39.350 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================================== 2023-09-30T00:21:39.350 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:21:39.350 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:21:39.350 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:21:39.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:21:39.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:21:39.352 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:39.352 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:21:39.352 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================================== 2023-09-30T00:21:39.352 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2023-09-30T00:21:39.352 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:39.353 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 1.9 M 2023-09-30T00:21:39.353 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 25 M 2023-09-30T00:21:39.353 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:21:39.557 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:21:39.618 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 77 kB/s | 16 kB 00:00 2023-09-30T00:21:39.618 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 156 kB/s | 33 kB 00:00 2023-09-30T00:21:39.625 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 217 kB/s | 47 kB 00:00 2023-09-30T00:21:39.759 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 579 kB/s | 81 kB 00:00 2023-09-30T00:21:39.830 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 859 kB/s | 61 kB 00:00 2023-09-30T00:21:39.884 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.8 MB/s | 155 kB 00:00 2023-09-30T00:21:39.905 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): python3-rsa-4.9-2.el8.noarch.rpm 229 kB/s | 64 kB 00:00 2023-09-30T00:21:39.919 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2023-09-30T00:21:39.946 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 655 kB/s | 27 kB 00:00 2023-09-30T00:21:39.951 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.9 MB/s | 126 kB 00:00 2023-09-30T00:21:39.979 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 3.4 MB/s | 110 kB 00:00 2023-09-30T00:21:39.983 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 640 kB/s | 20 kB 00:00 2023-09-30T00:21:40.049 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2023-09-30T00:21:40.049 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:40.049 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.8 MB/s | 1.9 MB 00:00 2023-09-30T00:21:40.055 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:21:40.069 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:21:40.069 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:21:40.155 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:21:40.160 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:21:40.693 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:21:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:21:41.102 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T00:21:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:21:41.273 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T00:21:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-09-30T00:21:41.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-09-30T00:21:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-09-30T00:21:41.987 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-09-30T00:21:42.273 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:21:42.578 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:21:42.713 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-09-30T00:21:42.713 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:20 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:42.807 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T00:21:43.023 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:21:43.282 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T00:21:44.434 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T00:21:44.742 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:21:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T00:21:44.850 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T00:21:44.851 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:44.852 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:21:44.915 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:21:44.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:21:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:21:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-09-30T00:21:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-09-30T00:21:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-09-30T00:21:45.377 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:46.316 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-09-30T00:21:46.316 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:26 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:46.376 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:24 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:48.262 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:21:48.279 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:21:48.281 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:48.282 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:21:48.283 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-09-30T00:21:48.283 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-09-30T00:21:48.283 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:48.283 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 1.9 M 2023-09-30T00:21:48.283 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 25 M 2023-09-30T00:21:48.283 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:21:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:21:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:21:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:21:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:21:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:21:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:48.365 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:48.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:21:48.540 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 214 kB/s | 47 kB 00:00 2023-09-30T00:21:48.611 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 56 kB/s | 16 kB 00:00 2023-09-30T00:21:48.645 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 101 kB/s | 33 kB 00:00 2023-09-30T00:21:48.737 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 412 kB/s | 81 kB 00:00 2023-09-30T00:21:48.746 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-rsa-4.9-2.el8.noarch.rpm 632 kB/s | 64 kB 00:00 2023-09-30T00:21:48.778 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 846 kB/s | 27 kB 00:00 2023-09-30T00:21:48.796 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 7.1 MB/s | 126 kB 00:00 2023-09-30T00:21:48.812 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-pyasn1-modules-0.3.7-6.el8.noar 6.5 MB/s | 110 kB 00:00 2023-09-30T00:21:48.828 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 1.3 MB/s | 20 kB 00:00 2023-09-30T00:21:48.832 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-websocket-client-0.56.0-5.el8. 645 kB/s | 61 kB 00:00 2023-09-30T00:21:48.845 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 9.0 MB/s | 155 kB 00:00 2023-09-30T00:21:48.871 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:48.871 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 1.1 MB/s | 43 kB 00:00 2023-09-30T00:21:49.108 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2023-09-30T00:21:49.108 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:49.109 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.3 MB/s | 1.9 MB 00:00 2023-09-30T00:21:49.115 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:21:49.130 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:21:49.131 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:21:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:21:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:29 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:49.857 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:21:50.179 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:21:50.456 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T00:21:50.729 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T00:21:51.037 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-09-30T00:21:51.286 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-09-30T00:21:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-09-30T00:21:51.556 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:21:51.557 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================================== 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================================== 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T00:21:51.558 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:21:51.559 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:51.559 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:21:51.559 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================================== 2023-09-30T00:21:51.559 INFO:teuthology.orchestra.run.smithi146.stdout:Install 5 Packages 2023-09-30T00:21:51.559 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:51.559 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.0 M 2023-09-30T00:21:51.560 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2023-09-30T00:21:51.560 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:21:51.803 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-09-30T00:21:52.030 INFO:teuthology.orchestra.run.smithi146.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 801 kB/s | 121 kB 00:00 2023-09-30T00:21:52.073 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T00:21:52.080 INFO:teuthology.orchestra.run.smithi146.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 300 kB/s | 60 kB 00:00 2023-09-30T00:21:52.151 INFO:teuthology.orchestra.run.smithi146.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 40 MB/s | 4.8 MB 00:00 2023-09-30T00:21:52.156 INFO:teuthology.orchestra.run.smithi146.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 6.9 MB/s | 538 kB 00:00 2023-09-30T00:21:52.234 INFO:teuthology.orchestra.run.smithi146.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.3 MB/s | 476 kB 00:00 2023-09-30T00:21:52.236 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:21:52.237 INFO:teuthology.orchestra.run.smithi146.stdout:Total 8.8 MB/s | 6.0 MB 00:00 2023-09-30T00:21:52.240 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:21:52.250 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:21:52.250 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:21:52.317 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:21:52.349 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:21:52.354 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:21:52.573 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T00:21:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:21:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T00:21:53.839 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T00:21:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T00:21:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T00:21:54.332 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:21:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-09-30T00:21:54.656 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-09-30T00:21:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-30T00:21:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:21:54.657 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-09-30T00:21:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:21:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:21:55.181 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:21:55.181 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:21:55.181 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:21:55.181 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:21:55.181 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T00:21:55.840 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-09-30T00:21:55.840 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:33 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:21:56.109 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T00:21:56.109 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:36 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:57.783 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:21:57.892 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:57.892 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:21:57.892 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:57.892 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:21:57.892 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:21:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:21:57.894 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:21:57.894 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:21:58.007 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:21:58.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:21:58.106 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:21:58.107 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:21:58.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2023-09-30T00:21:58.432 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:58.608 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:21:59.368 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:39 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:21:59.406 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:37 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:01.236 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:01.236 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2023-09-30T00:22:01.237 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:01.238 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 957 k 2023-09-30T00:22:01.238 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 3.0 M 2023-09-30T00:22:01.238 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:22:01.283 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2023-09-30T00:22:01.401 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-09-30T00:22:01.402 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:22:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-09-30T00:22:01.402 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2023-09-30T00:22:01.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.0 M 2023-09-30T00:22:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2023-09-30T00:22:01.404 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:22:01.453 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 4.0 MB/s | 874 kB 00:00 2023-09-30T00:22:01.454 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:01.454 INFO:teuthology.orchestra.run.smithi146.stdout:Total 4.3 MB/s | 957 kB 00:00 2023-09-30T00:22:01.456 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:22:01.477 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:22:01.477 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:22:01.562 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:22:01.567 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:22:01.587 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 795 kB/s | 121 kB 00:00 2023-09-30T00:22:01.636 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 299 kB/s | 60 kB 00:00 2023-09-30T00:22:01.690 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 46 MB/s | 4.8 MB 00:00 2023-09-30T00:22:01.696 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 8.9 MB/s | 538 kB 00:00 2023-09-30T00:22:01.727 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.6 MB/s | 476 kB 00:00 2023-09-30T00:22:01.729 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:01.730 INFO:teuthology.orchestra.run.smithi031.stdout:Total 18 MB/s | 6.0 MB 00:00 2023-09-30T00:22:01.733 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:22:01.738 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:22:01.744 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:22:01.744 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:22:01.842 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:22:01.846 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:22:02.067 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T00:22:02.271 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:22:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:22:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:22:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:22:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T00:22:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T00:22:03.926 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T00:22:03.926 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:43 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:22:04.275 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:22:04.484 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:22:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:22:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:22:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:22:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:22:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T00:22:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T00:22:05.830 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:43 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:05.896 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:22:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:05.994 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:22:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T00:22:05.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:05.994 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:06.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-volume 2023-09-30T00:22:06.516 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:07.284 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:07.804 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:22:07.913 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:07.914 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:08.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2023-09-30T00:22:08.424 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:09.122 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:09.123 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:09.123 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2023-09-30T00:22:09.123 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:09.123 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 267 k 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.3 M 2023-09-30T00:22:09.124 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:22:09.299 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-09-30T00:22:09.300 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:09.300 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-09-30T00:22:09.300 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:22:09.310 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:22:09.311 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:22:09.336 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:09.337 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:22:09.341 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:22:09.519 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:22:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:22:10.529 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:22:11.212 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:11.213 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:11.213 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:22:11.213 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:11.213 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:22:11.213 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2023-09-30T00:22:11.214 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:11.215 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 957 k 2023-09-30T00:22:11.215 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 3.0 M 2023-09-30T00:22:11.215 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:22:11.255 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.0 MB/s | 83 kB 00:00 2023-09-30T00:22:11.469 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:22:11.470 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:11.592 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 874 kB 00:00 2023-09-30T00:22:11.593 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:11.594 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.5 MB/s | 957 kB 00:00 2023-09-30T00:22:11.595 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:22:11.619 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:22:11.619 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:22:11.711 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:22:11.715 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:22:12.878 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:22:13.189 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T00:22:13.365 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:22:13.376 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:22:13.464 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:13.464 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:22:13.464 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:22:13.465 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:13.465 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:13.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2023-09-30T00:22:13.796 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:22:13.796 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:22:13.969 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:14.784 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:14.924 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T00:22:14.925 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:52 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:16.585 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:16.586 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:16.586 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 130 k 2023-09-30T00:22:16.587 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 464 k 2023-09-30T00:22:16.588 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:22:16.739 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 853 kB/s | 130 kB 00:00 2023-09-30T00:22:16.740 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:16.740 INFO:teuthology.orchestra.run.smithi146.stdout:Total 845 kB/s | 130 kB 00:00 2023-09-30T00:22:16.741 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:22:16.747 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:22:16.747 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:22:16.803 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:22:16.808 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:22:16.952 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:22:16.958 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:22:17.064 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:17.064 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:22:17.064 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T00:22:17.064 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:17.064 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:17.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2023-09-30T00:22:17.421 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:17.571 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:18.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:18.335 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:02:58 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:18.499 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:56 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:20.199 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:22:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:20.298 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:22:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:22:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:20.298 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:20.396 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:22:20.397 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:20.398 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 267 k 2023-09-30T00:22:20.398 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.3 M 2023-09-30T00:22:20.398 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:22:20.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2023-09-30T00:22:20.572 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-09-30T00:22:20.573 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:20.573 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-09-30T00:22:20.574 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:22:20.585 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:22:20.585 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:22:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:22:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:22:20.799 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:20.808 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:22:21.107 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:22:21.556 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:22:21.598 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:01 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:22.620 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:22:22.620 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:00 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:22.920 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:23.396 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:23.397 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:23.397 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:23.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2023-09-30T00:22:23.871 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:24.647 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:22:24.686 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:04 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:24.754 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:22:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:22:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:24.754 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:24.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2023-09-30T00:22:25.282 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:26.145 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:04 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:26.789 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:26.790 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:26.790 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-09-30T00:22:26.790 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:26.790 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:22:26.790 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:22:26.790 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:26.791 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:22:26.791 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:26.791 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:22:26.791 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:26.791 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 25 k 2023-09-30T00:22:26.792 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 109 k 2023-09-30T00:22:26.792 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:22:26.977 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 134 kB/s | 25 kB 00:00 2023-09-30T00:22:26.977 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:26.978 INFO:teuthology.orchestra.run.smithi146.stdout:Total 133 kB/s | 25 kB 00:00 2023-09-30T00:22:26.978 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:22:26.982 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:22:26.982 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:22:26.984 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:22:26.988 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:22:27.038 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:22:27.241 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:28.096 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:22:28.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 130 k 2023-09-30T00:22:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 464 k 2023-09-30T00:22:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:22:28.356 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 504 kB/s | 130 kB 00:00 2023-09-30T00:22:28.356 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:28.357 INFO:teuthology.orchestra.run.smithi031.stdout:Total 500 kB/s | 130 kB 00:00 2023-09-30T00:22:28.357 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:22:28.365 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:22:28.365 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:22:28.426 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:22:28.431 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:22:28.441 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:28.441 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:08 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:22:28.755 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:29.036 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:30.074 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:08 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:30.338 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:22:30.434 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:30.434 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:22:30.434 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:22:30.434 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:30.434 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:30.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2023-09-30T00:22:30.933 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:31.667 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:11 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:32.029 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:22:32.135 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:32.136 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:22:32.136 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:22:32.136 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:32.136 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:32.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2023-09-30T00:22:32.655 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:33.001 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:33.485 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:33.486 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:33.486 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:33.541 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:11 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:33.592 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2023-09-30T00:22:33.965 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:34.761 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:14 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:34.935 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:35.437 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:22:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:35.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2023-09-30T00:22:35.925 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:36.109 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:36.584 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:36.586 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:36.586 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:36.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2023-09-30T00:22:36.839 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:14 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:37.076 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:37.838 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:17 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:38.708 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:22:38.709 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 25 k 2023-09-30T00:22:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 109 k 2023-09-30T00:22:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:22:38.821 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 223 kB/s | 25 kB 00:00 2023-09-30T00:22:38.821 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:38.822 INFO:teuthology.orchestra.run.smithi031.stdout:Total 220 kB/s | 25 kB 00:00 2023-09-30T00:22:38.822 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:22:38.826 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:22:38.826 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:22:38.828 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:22:38.833 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:22:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:22:39.112 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:39.142 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:39.384 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:39.618 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:39.619 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:39.619 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:39.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2023-09-30T00:22:40.104 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:40.445 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:18 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:40.880 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:20 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:42.342 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:42.382 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:22:42.489 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:42.489 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:22:42.489 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:22:42.489 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:22:42.490 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:42.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2023-09-30T00:22:42.816 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:42.817 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:42.817 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:42.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2023-09-30T00:22:43.015 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:43.301 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:43.874 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:21 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:44.097 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:24 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:45.325 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:45.416 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:45.825 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:45.826 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:22:45.826 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:45.900 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:45.901 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:45.902 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:45.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2023-09-30T00:22:46.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2023-09-30T00:22:46.322 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:47.219 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:25 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:47.237 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:27 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:48.574 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:48.887 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:49.049 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:49.050 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2023-09-30T00:22:49.050 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:49.152 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2023-09-30T00:22:49.377 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:49.378 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:22:49.378 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:49.529 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:49.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2023-09-30T00:22:49.990 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:50.275 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:30 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:50.853 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:28 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:52.115 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:52.116 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:52.116 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2023-09-30T00:22:52.116 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:52.116 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 90 k 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 264 k 2023-09-30T00:22:52.117 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:22:52.255 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 663 kB/s | 90 kB 00:00 2023-09-30T00:22:52.255 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:52.255 INFO:teuthology.orchestra.run.smithi146.stdout:Total 656 kB/s | 90 kB 00:00 2023-09-30T00:22:52.255 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:22:52.266 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:22:52.266 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:22:52.323 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:22:52.327 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:22:52.340 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:22:52.670 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:52.845 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:52.847 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:22:52.847 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:52.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2023-09-30T00:22:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:53.346 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:22:53.928 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:33 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:54.229 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:32 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:55.673 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:55.772 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:22:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:55.869 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:22:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:22:55.869 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:55.869 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:22:55.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2023-09-30T00:22:56.174 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:56.175 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:22:56.175 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:56.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2023-09-30T00:22:56.370 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:56.663 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:22:57.121 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:37 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:22:57.522 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:35 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:22:58.920 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:58.921 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:22:58.922 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:22:58.922 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:22:58.922 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:22:58.922 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.5 M 2023-09-30T00:22:58.922 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 16 M 2023-09-30T00:22:58.922 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:22:58.981 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:22:59.392 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 7.3 MB/s | 3.5 MB 00:00 2023-09-30T00:22:59.393 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:22:59.394 INFO:teuthology.orchestra.run.smithi146.stdout:Total 7.3 MB/s | 3.5 MB 00:00 2023-09-30T00:22:59.394 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:22:59.407 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:22:59.407 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:22:59.475 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:22:59.477 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:22:59.479 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:22:59.479 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:22:59.480 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:22:59.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2023-09-30T00:22:59.939 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:22:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:23:00.129 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:00.593 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:00.875 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:38 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:01.521 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:01.521 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:41 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:23:02.289 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:23:02.794 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:23:02.795 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-30T00:23:02.795 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:23:02.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2023-09-30T00:23:03.295 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:23:03.369 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:23:03.465 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:03.465 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:23:03.465 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:23:03.466 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:03.466 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:23:03.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2023-09-30T00:23:03.959 INFO:teuthology.orchestra.run.smithi146.stdout:Updating Subscription Management repositories. 2023-09-30T00:23:04.199 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:42 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:04.702 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:44 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:23:06.118 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:23:06.118 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:06.119 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 90 k 2023-09-30T00:23:06.120 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 264 k 2023-09-30T00:23:06.120 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:23:06.251 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 686 kB/s | 90 kB 00:00 2023-09-30T00:23:06.251 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:23:06.252 INFO:teuthology.orchestra.run.smithi031.stdout:Total 678 kB/s | 90 kB 00:00 2023-09-30T00:23:06.252 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:23:06.265 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:23:06.265 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:23:06.324 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:23:06.328 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:23:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:23:06.547 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2023-09-30T00:23:06.547 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2023-09-30T00:23:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:06.549 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 173 k 2023-09-30T00:23:06.549 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 552 k 2023-09-30T00:23:06.549 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2023-09-30T00:23:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:06.860 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 557 kB/s | 173 kB 00:00 2023-09-30T00:23:06.860 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:23:06.860 INFO:teuthology.orchestra.run.smithi146.stdout:Total 555 kB/s | 173 kB 00:00 2023-09-30T00:23:06.861 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2023-09-30T00:23:06.874 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2023-09-30T00:23:06.874 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2023-09-30T00:23:06.931 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2023-09-30T00:23:06.936 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2023-09-30T00:23:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2023-09-30T00:23:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:07.523 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:08.075 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:08.075 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:46 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:08.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:08.458 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:03:48 ago on Sat 30 Sep 2023 12:19:20 AM UTC. 2023-09-30T00:23:10.100 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:23:10.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:10.207 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:23:10.207 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:23:10.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:10.207 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:23:10.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-09-30T00:23:10.360 INFO:teuthology.orchestra.run.smithi146.stdout:Installed products updated. 2023-09-30T00:23:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:10.460 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2023-09-30T00:23:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:23:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:10.460 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2023-09-30T00:23:10.584 DEBUG:teuthology.parallel:result is None 2023-09-30T00:23:10.716 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:23:11.576 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:49 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:13.454 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:23:13.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:13.456 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.5 M 2023-09-30T00:23:13.456 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 16 M 2023-09-30T00:23:13.456 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:23:14.200 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 4.6 MB/s | 3.5 MB 00:00 2023-09-30T00:23:14.201 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:23:14.202 INFO:teuthology.orchestra.run.smithi031.stdout:Total 4.6 MB/s | 3.5 MB 00:00 2023-09-30T00:23:14.202 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:23:14.218 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:23:14.218 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:23:14.293 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:23:14.298 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:23:14.786 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:23:14.940 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:16.531 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:16.531 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:54 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:18.531 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:23:18.638 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:18.638 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:23:18.639 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:23:18.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:18.639 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:23:18.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2023-09-30T00:23:19.190 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-09-30T00:23:20.101 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:58 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:22.008 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-30T00:23:22.009 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-30T00:23:22.010 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:22.010 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 173 k 2023-09-30T00:23:22.010 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 552 k 2023-09-30T00:23:22.010 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-30T00:23:22.163 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2023-09-30T00:23:22.163 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:23:22.164 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.1 MB/s | 173 kB 00:00 2023-09-30T00:23:22.164 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-30T00:23:22.178 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-30T00:23:22.178 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-30T00:23:22.239 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-30T00:23:22.243 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-30T00:23:22.384 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-30T00:23:22.581 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:22.861 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:23:23.966 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:01 ago on Sat 30 Sep 2023 12:19:22 AM UTC. 2023-09-30T00:23:25.942 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-09-30T00:23:26.049 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:26.049 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-30T00:23:26.049 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:23:26.049 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:26.049 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-30T00:23:26.159 DEBUG:teuthology.parallel:result is None 2023-09-30T00:23:26.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:23:26.318 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:23:26.339 INFO:teuthology.orchestra.run.smithi031.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:23:26.339 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:23:26.339 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:23:26.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:23:26.614 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:23:26.636 INFO:teuthology.orchestra.run.smithi146.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:23:26.636 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:23:26.636 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:23:26.636 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:23:26.637 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:26.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:23:26.664 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:26.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:23:26.704 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:23:26.704 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:26.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:23:26.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:23:26.796 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:26.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:23:26.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:23:26.888 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:23:26.888 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:26.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:23:26.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:23:26.978 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:26.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:23:27.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:23:27.068 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:23:27.069 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:27.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:23:27.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:23:27.160 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:27.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:23:27.187 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:23:27.251 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:23:27.307 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:23:27.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-09-30T00:23:27.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2023-09-30T00:23:27.334 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:23:27.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:23:27.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:23:27.401 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:23:27.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:23:27.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:23:27.479 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:23:27.479 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:23:27.480 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406696', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbLSWpInlss1NhwAmRhZO8PmH0HZUnnVKkzJwG9jwzE9fx6pDf+J6C0HPNEsaoc1Uc10f1MWySqmXKWh7mnFgE=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCw9suGSXpcUQH2x096mCSKwsSx8zzV4kALqY6WlEzmOhUXQvknAtq3ZjgAPq4f1J85RnSx+ZyY6CVcP57h+Fsw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250', 'sudo ceph osd pool set cache min_read_recency_for_promote 2', 'sudo ceph osd pool set cache min_write_recency_for_promote 2']}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:23:27.480 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:23:27.511 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:23:27.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:23:27.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:23:27.590 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:27.590 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:23:27.646 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:23:27.646 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-09-30T00:23:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:23:27.702 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:27.702 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 216690 Links: 1 2023-09-30T00:23:27.702 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:27.703 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:27.703 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-30 00:23:15.142846179 +0000 2023-09-30T00:23:27.703 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-30 00:18:39.414623247 +0000 2023-09-30T00:23:27.703 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-30 00:18:39.414623247 +0000 2023-09-30T00:23:27.703 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-09-30T00:23:27.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:23:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-30T00:23:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-30T00:23:27.767 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000254481 s, 2.0 MB/s 2023-09-30T00:23:27.768 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:23:27.825 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-09-30T00:23:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:23:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:27.881 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 217655 Links: 1 2023-09-30T00:23:27.881 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:27.882 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:27.882 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-30 00:23:15.142846179 +0000 2023-09-30T00:23:27.882 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-30 00:18:39.884615850 +0000 2023-09-30T00:23:27.882 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-30 00:18:39.884615850 +0000 2023-09-30T00:23:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-09-30T00:23:27.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:23:27.946 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-30T00:23:27.946 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-30T00:23:27.946 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000223712 s, 2.3 MB/s 2023-09-30T00:23:27.947 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:23:28.004 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 216831 Links: 1 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-30 00:23:15.142846179 +0000 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-30 00:18:40.384607981 +0000 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-30 00:18:40.384607981 +0000 2023-09-30T00:23:28.059 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-09-30T00:23:28.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:23:28.123 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-30T00:23:28.123 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-30T00:23:28.123 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000227025 s, 2.3 MB/s 2023-09-30T00:23:28.124 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:23:28.180 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-09-30T00:23:28.236 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:23:28.236 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:28.236 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 214755 Links: 1 2023-09-30T00:23:28.236 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:28.236 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:28.236 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-30 00:23:15.142846179 +0000 2023-09-30T00:23:28.237 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-30 00:18:40.883600128 +0000 2023-09-30T00:23:28.237 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-30 00:18:40.883600128 +0000 2023-09-30T00:23:28.237 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-09-30T00:23:28.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:23:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-30T00:23:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-30T00:23:28.300 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000180875 s, 2.8 MB/s 2023-09-30T00:23:28.301 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:23:28.357 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:28.357 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:28.357 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:23:28.372 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:23:28.373 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 218511 Links: 1 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-09-30 00:23:00.361664133 +0000 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-09-30 00:18:23.950744966 +0000 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-09-30 00:18:23.950744966 +0000 2023-09-30T00:23:28.428 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-09-30T00:23:28.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:23:28.491 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-09-30T00:23:28.491 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-09-30T00:23:28.491 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000267194 s, 1.9 MB/s 2023-09-30T00:23:28.492 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:23:28.548 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 216804 Links: 1 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-09-30 00:23:00.362664119 +0000 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-09-30 00:18:24.431737789 +0000 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-09-30 00:18:24.431737789 +0000 2023-09-30T00:23:28.603 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-09-30T00:23:28.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:23:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-09-30T00:23:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-09-30T00:23:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00022323 s, 2.3 MB/s 2023-09-30T00:23:28.666 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:23:28.722 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2023-09-30T00:23:28.776 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:23:28.776 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 214809 Links: 1 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-09-30 00:23:00.362664119 +0000 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-09-30 00:18:24.913730596 +0000 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-09-30 00:18:24.913730596 +0000 2023-09-30T00:23:28.777 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-09-30T00:23:28.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:23:28.839 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-09-30T00:23:28.840 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-09-30T00:23:28.840 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000199103 s, 2.6 MB/s 2023-09-30T00:23:28.841 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:23:28.896 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2023-09-30T00:23:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 214857 Links: 1 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2023-09-30 00:23:00.362664119 +0000 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2023-09-30 00:18:25.431722867 +0000 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2023-09-30 00:18:25.431722867 +0000 2023-09-30T00:23:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2023-09-30T00:23:28.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:23:29.014 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2023-09-30T00:23:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2023-09-30T00:23:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000227613 s, 2.2 MB/s 2023-09-30T00:23:29.016 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:23:29.071 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:29.071 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:23:29.071 INFO:tasks.ceph:Generating config... 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] bdev async discard = True 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T00:23:29.073 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] osd backoff on peering = True 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T00:23:29.074 INFO:tasks.ceph:[osd] osd delete sleep = 1 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd max backfills = 3 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd max scrubs = 2 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-09-30T00:23:29.075 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-09-30T00:23:29.076 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T00:23:29.076 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2023-09-30T00:23:29.076 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2023-09-30T00:23:29.076 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] ms inject delay max = 1 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2023-09-30T00:23:29.076 INFO:tasks.ceph:[global] ms inject delay type = osd 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] ms type = async 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_recovery_max_active = 10 2023-09-30T00:23:29.077 INFO:tasks.ceph:[global] osd_recovery_max_single_start = 10 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T00:23:29.078 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T00:23:29.079 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:23:29.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:23:29.128 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:23:29.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:23:29.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:23:29.237 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31'), ('mon.c', '[v2:172.21.15.31:3301,v1:172.21.15.31:6790]'), ('mon.b', '172.21.15.146')] 2023-09-30T00:23:29.238 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.146', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T00:23:29.239 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:29.239 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:23:29.292 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --addv c '[v2:172.21.15.31:3301,v1:172.21.15.31:6790]' --add b 172.21.15.146 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:29.375 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:29.375 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 38d43cc8-1041-4ca0-9db0-4fe8f50adae4 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = octopus 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 38d43cc8-1041-4ca0-9db0-4fe8f50adae4 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-09-30T00:23:29.376034+0000 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-09-30T00:23:29.376034+0000 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 15 (octopus) 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 3 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:2: [v2:172.21.15.31:3301/0,v1:172.21.15.31:6790/0] mon.c 2023-09-30T00:23:29.376 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T00:23:29.377 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:23:29.432 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 38d43cc8-1041-4ca0-9db0-4fe8f50adae4... 2023-09-30T00:23:29.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:23:29.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-09-30T00:23:29.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:23:29.518 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:23:29.519 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-09-30T00:23:29.520 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = "172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.146" 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: mon election default strategy = 3 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms bind msgr1 = False 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms bind msgr2 = True 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay max = 1 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay probability = 0.005 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay type = osd 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject internal delays = 0.002 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 2500 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: ms type = async 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_pg_log_entries = 2 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: osd_min_pg_log_entries = 1 2023-09-30T00:23:29.521 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pg_log_trim_min = 0 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_min_size = 2 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_size = 3 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd_recovery_max_active = 10 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd_recovery_max_single_start = 10 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 38d43cc8-1041-4ca0-9db0-4fe8f50adae4 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-09-30T00:23:29.522 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 120 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-09-30T00:23:29.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-09-30T00:23:29.524 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 20 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: bdev async discard = True 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: bdev enable discard = True 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = stupid 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore zero block detection = True 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 20 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 20 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 10 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:23:29.525 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd backoff on peering = True 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug reject backfill probability = 0.3 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify cached snaps = True 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify missing on start = True 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd delete sleep = 1 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd max backfills = 3 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd max markdown count = 1000 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd max scrubs = 2 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock override recovery settings = True 2023-09-30T00:23:29.526 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd op queue = debug_random 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd op queue cut off = debug_random 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub min interval = 60 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd snap trim sleep = 2 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_pg_log_entries = 3000 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: osd_min_pg_log_entries = 3000 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-09-30T00:23:29.527 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-09-30T00:23:29.528 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:23:29.529 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon min osdmap epochs = 50 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon osdmap full prune interval = 2 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon osdmap full prune min = 15 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: mon scrub interval = 300 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: paxos service trim min = 10 2023-09-30T00:23:29.530 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.c] 2023-09-30T00:23:29.531 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.b] 2023-09-30T00:23:29.532 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.533 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.534 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:23:29.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.31,[v2:172.21.15.31:3301,v1:172.21.15.31:6790],172.21.15.146" 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: mon election default strategy = 3 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: ms bind msgr1 = False 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: ms bind msgr2 = True 2023-09-30T00:23:29.536 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject delay max = 1 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject delay probability = 0.005 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject delay type = osd 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject internal delays = 0.002 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 2500 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: ms type = async 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_pg_log_entries = 2 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_min_pg_log_entries = 1 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pg_log_trim_min = 0 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_min_size = 2 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_size = 3 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_recovery_max_active = 10 2023-09-30T00:23:29.537 INFO:teuthology.orchestra.run.smithi146.stdout: osd_recovery_max_single_start = 10 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 38d43cc8-1041-4ca0-9db0-4fe8f50adae4 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 120 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2023-09-30T00:23:29.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: debug monc = 20 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = stupid 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2023-09-30T00:23:29.540 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore zero block detection = True 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 20 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 20 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 10 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: osd backoff on peering = True 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug reject backfill probability = 0.3 2023-09-30T00:23:29.541 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify cached snaps = True 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify missing on start = True 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd delete sleep = 1 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd max backfills = 3 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd max markdown count = 1000 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd max scrubs = 2 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock override recovery settings = True 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue = debug_random 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue cut off = debug_random 2023-09-30T00:23:29.542 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub min interval = 60 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: osd shutdown pgref assert = True 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: osd snap trim sleep = 2 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: osd_max_pg_log_entries = 3000 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: osd_min_pg_log_entries = 3000 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:23:29.543 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:23:29.544 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:23:29.545 INFO:teuthology.orchestra.run.smithi146.stdout: mon min osdmap epochs = 50 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune interval = 2 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune min = 15 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: mon scrub interval = 300 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: paxos service trim min = 10 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:23:29.546 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:23:29.547 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2023-09-30T00:23:29.547 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2023-09-30T00:23:29.547 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2023-09-30T00:23:29.547 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:23:29.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:23:29.624 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:23:29.624 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:29.625 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:23:29.639 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:29.639 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:23:29.694 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:23:29.694 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:29.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:23:29.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:23:29.769 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:29.769 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:29.825 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:23:29.825 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:29.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:23:29.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:23:29.860 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:23:29.860 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:29.914 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:23:29.914 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:23:29.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:23:29.963 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:23:29.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:23:30.032 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:23:30.034 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:23:30.035 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:23:30.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:23:30.074 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:23:30.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T00:23:30.125 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T00:23:30.137 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:23:30.138 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:23:30.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:23:30.163 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:30.163 INFO:tasks.ceph:role: osd.0 2023-09-30T00:23:30.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:23:30.163 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:23:30.234 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:30.234 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:30.235 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:30.667 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-30T00:23:30.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-30T00:23:30.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:23:30.830 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:23:30.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:23:30.930 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:30.930 INFO:tasks.ceph:role: osd.1 2023-09-30T00:23:30.930 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:23:30.930 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:31.003 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:31.004 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:31.004 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:31.423 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-30T00:23:31.425 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-30T00:23:31.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T00:23:31.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:23:31.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:23:31.629 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:31.629 INFO:tasks.ceph:role: osd.2 2023-09-30T00:23:31.629 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:23:31.630 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:23:31.702 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:31.702 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:31.702 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:31.703 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:32.153 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-30T00:23:32.155 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-30T00:23:32.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T00:23:32.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:23:32.342 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:23:32.409 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:32.409 INFO:tasks.ceph:role: osd.3 2023-09-30T00:23:32.409 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:23:32.409 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:23:32.483 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:32.483 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:32.483 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:32.483 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 2023-09-30T00:23:32.484 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:32.484 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:32.484 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:32.484 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:32.484 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:32.484 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:32.920 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-30T00:23:32.922 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-30T00:23:32.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T00:23:33.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:23:33.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:33.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.233+0000 7f27c805a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:23:33.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.233+0000 7f27c805a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:23:33.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.233+0000 7f27c805a540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:23:33.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.233+0000 7f27c805a540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:23:33.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.233+0000 7f27c805a540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:23:33.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.234+0000 7f27c805a540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T00:23:33.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:33.757+0000 7f27c805a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:34.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:34.522+0000 7f27c805a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:35.284+0000 7f27c805a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:36.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:23:36.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.196+0000 7f121a526540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:23:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.196+0000 7f121a526540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:23:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.196+0000 7f121a526540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:23:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.196+0000 7f121a526540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:23:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.196+0000 7f121a526540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:23:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.197+0000 7f121a526540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T00:23:36.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:36.720+0000 7f121a526540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:37.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:37.483+0000 7f121a526540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:38.247+0000 7f121a526540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:39.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:23:39.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.157+0000 7f3f3c88b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:23:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.157+0000 7f3f3c88b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:23:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.157+0000 7f3f3c88b540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:23:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.158+0000 7f3f3c88b540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:23:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.158+0000 7f3f3c88b540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:23:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.158+0000 7f3f3c88b540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T00:23:39.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:39.684+0000 7f3f3c88b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:40.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:40.455+0000 7f3f3c88b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:41.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:41.217+0000 7f3f3c88b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:42.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:23:42.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:42.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.130+0000 7f9c85685540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:23:42.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.131+0000 7f9c85685540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:23:42.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.131+0000 7f9c85685540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:23:42.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.131+0000 7f9c85685540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:23:42.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.131+0000 7f9c85685540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:23:42.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.131+0000 7f9c85685540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T00:23:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:42.654+0000 7f9c85685540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:43.421+0000 7f9c85685540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:44.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-30T00:23:44.182+0000 7f9c85685540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:44.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:23:45.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T00:23:45.038 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:45.039 INFO:tasks.ceph:role: osd.4 2023-09-30T00:23:45.039 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:23:45.039 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:23:45.111 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:45.111 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:45.112 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:45.543 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-09-30T00:23:45.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-09-30T00:23:45.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T00:23:45.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T00:23:45.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T00:23:45.787 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:45.787 INFO:tasks.ceph:role: osd.5 2023-09-30T00:23:45.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:23:45.787 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:23:45.858 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:45.858 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:45.859 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:46.291 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-09-30T00:23:46.293 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-09-30T00:23:46.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T00:23:46.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T00:23:46.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T00:23:46.492 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:46.492 INFO:tasks.ceph:role: osd.6 2023-09-30T00:23:46.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:23:46.492 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:46.564 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:46.565 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:46.565 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:47.010 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-09-30T00:23:47.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-09-30T00:23:47.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T00:23:47.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T00:23:47.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T00:23:47.257 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:23:47.257 INFO:tasks.ceph:role: osd.7 2023-09-30T00:23:47.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:23:47.257 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:23:47.328 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:23:47.329 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:23:47.329 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:23:47.765 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2023-09-30T00:23:47.767 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2023-09-30T00:23:47.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T00:23:47.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T00:23:47.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.045+0000 7f142b2bc540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T00:23:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.045+0000 7f142b2bc540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T00:23:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.045+0000 7f142b2bc540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:23:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.045+0000 7f142b2bc540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:23:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.045+0000 7f142b2bc540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:23:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.046+0000 7f142b2bc540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T00:23:48.569 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:48.568+0000 7f142b2bc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:49.339 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:49.337+0000 7f142b2bc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:50.101 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:50.100+0000 7f142b2bc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:50.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T00:23:50.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:51.004 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.002+0000 7f1d6a624540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T00:23:51.004 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.003+0000 7f1d6a624540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T00:23:51.004 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.003+0000 7f1d6a624540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:23:51.004 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.003+0000 7f1d6a624540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:23:51.004 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.003+0000 7f1d6a624540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:23:51.005 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.003+0000 7f1d6a624540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T00:23:51.526 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:51.524+0000 7f1d6a624540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:52.289 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:52.287+0000 7f1d6a624540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:53.052 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.051+0000 7f1d6a624540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:53.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T00:23:53.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:53.997 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.995+0000 7f2824647540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T00:23:53.998 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.996+0000 7f2824647540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T00:23:53.998 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.996+0000 7f2824647540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:23:53.998 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.996+0000 7f2824647540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:23:53.998 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.996+0000 7f2824647540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:23:53.998 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:53.996+0000 7f2824647540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T00:23:54.521 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:54.520+0000 7f2824647540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:55.285 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:55.284+0000 7f2824647540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.047+0000 7f2824647540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:56.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T00:23:56.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:23:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.958+0000 7f1c1edde540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T00:23:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.959+0000 7f1c1edde540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T00:23:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.959+0000 7f1c1edde540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:23:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.959+0000 7f1c1edde540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:23:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.959+0000 7f1c1edde540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:23:56.961 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:56.959+0000 7f1c1edde540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T00:23:57.485 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:57.484+0000 7f1c1edde540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:58.248 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:58.247+0000 7f1c1edde540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:59.010 INFO:teuthology.orchestra.run.smithi146.stderr:2023-09-30T00:23:59.009+0000 7f1c1edde540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:23:59.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T00:23:59.872 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:23:59.873 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:59.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T00:23:59.901 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:59.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:23:59.969 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:23:59.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:24:00.036 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:00.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:24:00.103 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:00.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:24:00.170 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:00.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:24:00.197 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:00.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T00:24:00.263 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:00.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T00:24:00.329 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:00.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T00:24:00.394 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:00.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T00:24:00.459 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:00.459 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:24:00.475 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:00.475 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T00:24:00.514 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:24:00.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:24:00.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:24:00.557 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.y] 2023-09-30T00:24:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCBahdlAxN6ORAAvcu+b0ryXCpBvCU9L03dmA== 2023-09-30T00:24:00.557 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-09-30T00:24:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCFahdlaPD2DRAArFx+PRhei8M27KeIerX0YA== 2023-09-30T00:24:00.557 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-09-30T00:24:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCIahdl/SbHCxAAsGjhD+63hvwB91XkUlW4rw== 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCLahdlHZN0CRAA2kGCXVwyOOmKAKUwdawNqA== 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCOahdlrg3aBxAAY2HxRr4RnxUsWKpDpRqyFA== 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCCahdleFrtARAAqAZ3up3AAnulSQ3PKpNnYg== 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.4] 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCUahdlVrrHAhAAHE8AtWu8OIjmn/wGPwI/pw== 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.5] 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCXahdlHZc/ABAAZH4Brid8/8zS07s1Zu65sA== 2023-09-30T00:24:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.6] 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCZahdlAqVxOxAA86zVr+ltok5cc2kM/gvfuA== 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.7] 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCcahdl2+w5ORAAo0fUX037kfdeouyJ2lWAWQ== 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCCahdlJ915BBAA8Gf5jANxEjF1KZEIIey43A== 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2023-09-30T00:24:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCCahdl/g99BxAAoILBUvdF1CWNHD8xWA2ivA== 2023-09-30T00:24:00.577 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2023-09-30T00:24:00.577 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCBahdlAxN6ORAAvcu+b0ryXCpBvCU9L03dmA== 2023-09-30T00:24:00.577 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCFahdlaPD2DRAArFx+PRhei8M27KeIerX0YA== 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCIahdl/SbHCxAAsGjhD+63hvwB91XkUlW4rw== 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCLahdlHZN0CRAA2kGCXVwyOOmKAKUwdawNqA== 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCOahdlrg3aBxAAY2HxRr4RnxUsWKpDpRqyFA== 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCCahdleFrtARAAqAZ3up3AAnulSQ3PKpNnYg== 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCUahdlVrrHAhAAHE8AtWu8OIjmn/wGPwI/pw== 2023-09-30T00:24:00.578 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCXahdlHZc/ABAAZH4Brid8/8zS07s1Zu65sA== 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCZahdlAqVxOxAA86zVr+ltok5cc2kM/gvfuA== 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCcahdl2+w5ORAAo0fUX037kfdeouyJ2lWAWQ== 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCCahdlJ915BBAA8Gf5jANxEjF1KZEIIey43A== 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2023-09-30T00:24:00.579 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCCahdl/g99BxAAoILBUvdF1CWNHD8xWA2ivA== 2023-09-30T00:24:00.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:24:00.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:24:00.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:24:00.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:24:00.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:00.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:01.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:01.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:01.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:01.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:24:01.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:24:01.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:24:01.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:24:01.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:24:01.218 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:24:01.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:24:01.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:24:01.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:24:01.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T00:24:01.425 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:24:01.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T00:24:01.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T00:24:01.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:24:01.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T00:24:01.685 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:24:01.688 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:24:01.740 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:24:01.740 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:24:01.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T00:24:01.742 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:24:01.743 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T00:24:01.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T00:24:01.744 INFO:tasks.ceph.mon.c:Started 2023-09-30T00:24:01.745 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T00:24:01.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T00:24:01.782 INFO:tasks.ceph.mon.b:Started 2023-09-30T00:24:01.782 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:24:01.783 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T00:24:01.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T00:24:01.785 INFO:tasks.ceph.mgr.y:Started 2023-09-30T00:24:01.785 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:24:01.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T00:24:01.787 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:24:01.787 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:24:01.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:24:08.232 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-09-30T00:24:08.244 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:24:08.244 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:08.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:24:08.269 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:08.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:24:08.334 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:08.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:24:08.399 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-30T00:24:08.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:24:08.464 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:08.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T00:24:08.489 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:08.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T00:24:08.554 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:08.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T00:24:08.618 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2023-09-30T00:24:08.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T00:24:08.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 856896a0-c275-4afb-9fc8-0ee850f2ef48 0 2023-09-30T00:24:08.952 INFO:teuthology.orchestra.run.smithi146.stdout:0 2023-09-30T00:24:08.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new e4e24806-a2e7-47e0-b0bf-8efce59fd6ce 1 2023-09-30T00:24:09.179 INFO:teuthology.orchestra.run.smithi146.stdout:1 2023-09-30T00:24:09.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new c05d81e7-d71d-4b9d-8990-2c33c75bc836 2 2023-09-30T00:24:09.368 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:09.367+0000 7fc61e282180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:24:09.374 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:09.373+0000 7fda4b5bf180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:24:09.404 INFO:teuthology.orchestra.run.smithi146.stdout:2 2023-09-30T00:24:09.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49 3 2023-09-30T00:24:09.508 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:09.506+0000 7fc61e282180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:24:09.514 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:09.513+0000 7fda4b5bf180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:24:09.587 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:09.585+0000 7fc61e282180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:24:09.593 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:09.592+0000 7fda4b5bf180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:24:09.632 INFO:teuthology.orchestra.run.smithi146.stdout:3 2023-09-30T00:24:09.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 61ec23b4-115e-41a9-a06b-be2c06e89ba7 4 2023-09-30T00:24:09.780 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:09.778+0000 7fc61e282180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:24:09.788 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:09.787+0000 7fda4b5bf180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:24:09.861 INFO:teuthology.orchestra.run.smithi146.stdout:4 2023-09-30T00:24:09.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 3ed6d497-61de-4612-bc24-8eb8052076a2 5 2023-09-30T00:24:09.961 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:09.960+0000 7fc61e282180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:24:09.969 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:09.968+0000 7fda4b5bf180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:24:10.042 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:10.040+0000 7fc61e282180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:24:10.050 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:10.049+0000 7fda4b5bf180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:24:10.087 INFO:teuthology.orchestra.run.smithi146.stdout:5 2023-09-30T00:24:10.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new b3c1cca2-1530-4e60-a4f0-3bedd6f433eb 6 2023-09-30T00:24:10.183 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:10.181+0000 7fc61e282180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:24:10.193 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:10.192+0000 7fda4b5bf180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:24:10.283 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:10.281+0000 7fc61e282180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:24:10.292 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:10.291+0000 7fda4b5bf180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:24:10.318 INFO:teuthology.orchestra.run.smithi146.stdout:6 2023-09-30T00:24:10.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 46762b2d-2b3b-4a90-8577-3b9278c07650 7 2023-09-30T00:24:10.544 INFO:teuthology.orchestra.run.smithi146.stdout:7 2023-09-30T00:24:10.557 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:24:10.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T00:24:10.560 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:24:10.560 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:24:10.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T00:24:10.562 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:24:10.562 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:24:10.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T00:24:10.564 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:24:10.564 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:24:10.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T00:24:10.566 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:24:10.566 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T00:24:10.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T00:24:10.568 INFO:tasks.ceph.osd.4:Started 2023-09-30T00:24:10.568 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T00:24:10.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T00:24:10.570 INFO:tasks.ceph.osd.5:Started 2023-09-30T00:24:10.573 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T00:24:10.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-30T00:24:10.575 INFO:tasks.ceph.osd.6:Started 2023-09-30T00:24:10.575 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T00:24:10.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-30T00:24:10.577 INFO:tasks.ceph.osd.7:Started 2023-09-30T00:24:10.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:24:10.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:10.807 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:24:10.540428+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:24:10.820 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:24:10.820 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:24:10.838 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:10.836+0000 7fc61e282180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:24:10.841 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:10.840+0000 7fda4b5bf180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:24:10.919 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:10.918+0000 7fda4b5bf180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:24:10.919 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:10.918+0000 7fc61e282180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:24:10.998 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:10.997+0000 7fda4b5bf180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:24:11.000 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:10.999+0000 7fc61e282180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:24:11.164 INFO:tasks.ceph.osd.5.smithi146.stderr:2023-09-30T00:24:11.162+0000 7f92d8adc540 -1 Falling back to public interface 2023-09-30T00:24:11.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-09-30T00:24:11.162+0000 7f049790e540 -1 Falling back to public interface 2023-09-30T00:24:11.166 INFO:tasks.ceph.osd.4.smithi146.stderr:2023-09-30T00:24:11.165+0000 7f3890d94540 -1 Falling back to public interface 2023-09-30T00:24:11.182 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-09-30T00:24:11.180+0000 7f210155d540 -1 Falling back to public interface 2023-09-30T00:24:11.190 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-09-30T00:24:11.188+0000 7f4fe0c59540 -1 Falling back to public interface 2023-09-30T00:24:11.197 INFO:tasks.ceph.osd.6.smithi146.stderr:2023-09-30T00:24:11.196+0000 7f91721f3540 -1 Falling back to public interface 2023-09-30T00:24:11.199 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-09-30T00:24:11.197+0000 7f737e5e2540 -1 Falling back to public interface 2023-09-30T00:24:11.203 INFO:tasks.ceph.osd.7.smithi146.stderr:2023-09-30T00:24:11.202+0000 7f2bb4b93540 -1 Falling back to public interface 2023-09-30T00:24:11.581 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:11.580+0000 7fda4b5bf180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:24:11.613 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:11.612+0000 7fc61e282180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:24:11.626 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:24:11.718 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:11.717+0000 7fda4b5bf180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:24:11.760 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:11.759+0000 7fc61e282180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:24:11.818 INFO:teuthology.misc.health.smithi031.stdout: 2023-09-30T00:24:11.818 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:24:10.540428+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:24:11.821 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:11.820+0000 7fda4b5bf180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:24:11.829 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:24:11.861 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:11.859+0000 7fc61e282180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:24:12.072 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:12.071+0000 7fda4b5bf180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:24:12.119 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:12.117+0000 7fc61e282180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:24:12.150 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:12.148+0000 7fda4b5bf180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:24:12.201 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:12.199+0000 7fc61e282180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:24:12.985 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:12.984+0000 7fda4b5bf180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:24:13.058 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.057+0000 7fda4b5bf180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:24:13.065 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.064+0000 7fc61e282180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:24:13.134 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.133+0000 7fda4b5bf180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:24:13.143 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.142+0000 7fc61e282180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:24:13.221 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.219+0000 7fc61e282180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:24:13.268 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.267+0000 7fda4b5bf180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:24:13.354 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.353+0000 7fda4b5bf180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:24:13.362 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.360+0000 7fc61e282180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:24:13.457 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.456+0000 7fc61e282180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:24:13.506 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.505+0000 7fda4b5bf180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:24:13.583 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.581+0000 7fda4b5bf180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:24:13.617 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.616+0000 7fc61e282180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:24:13.664 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.663+0000 7fda4b5bf180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:24:13.703 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.701+0000 7fc61e282180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:24:13.740 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.738+0000 7fda4b5bf180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:24:13.787 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.786+0000 7fd9db8b1700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:24:13.787 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.786+0000 7fd9db8b1700 -1 pg_autoscaler.serve: 2023-09-30T00:24:13.788 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.786+0000 7fc61e282180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:13.786+0000 7fd9db8b1700 -1 Traceback (most recent call last): 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr: self._maybe_adjust() 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr: if self.noautoscale: 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:24:13.789 INFO:tasks.ceph.mgr.x.smithi146.stderr: 2023-09-30T00:24:13.871 INFO:tasks.ceph.mgr.y.smithi031.stderr:2023-09-30T00:24:13.869+0000 7fc61e282180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:24:14.422 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-09-30T00:24:14.420+0000 7f049790e540 -1 osd.1 0 log_to_monitors true 2023-09-30T00:24:14.646 INFO:tasks.ceph.osd.5.smithi146.stderr:2023-09-30T00:24:14.645+0000 7f92d8adc540 -1 osd.5 0 log_to_monitors true 2023-09-30T00:24:14.673 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-09-30T00:24:14.672+0000 7f210155d540 -1 osd.0 0 log_to_monitors true 2023-09-30T00:24:14.696 INFO:tasks.ceph.osd.6.smithi146.stderr:2023-09-30T00:24:14.694+0000 7f91721f3540 -1 osd.6 0 log_to_monitors true 2023-09-30T00:24:14.707 INFO:tasks.ceph.osd.4.smithi146.stderr:2023-09-30T00:24:14.706+0000 7f3890d94540 -1 osd.4 0 log_to_monitors true 2023-09-30T00:24:14.711 INFO:tasks.ceph.osd.7.smithi146.stderr:2023-09-30T00:24:14.710+0000 7f2bb4b93540 -1 osd.7 0 log_to_monitors true 2023-09-30T00:24:14.724 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-09-30T00:24:14.722+0000 7f4fe0c59540 -1 osd.2 0 log_to_monitors true 2023-09-30T00:24:14.727 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-09-30T00:24:14.725+0000 7f737e5e2540 -1 osd.3 0 log_to_monitors true 2023-09-30T00:24:14.770 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:14.769+0000 7fda049c2700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T00:24:14.771 INFO:tasks.ceph.mgr.x.smithi146.stderr:2023-09-30T00:24:14.770+0000 7fda049c2700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T00:24:15.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-09-30T00:24:15.841+0000 7f048e52f700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:24:15.844 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-09-30T00:24:15.842+0000 7f7376a06700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:24:15.844 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-09-30T00:24:15.842+0000 7f20f817e700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:24:15.844 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-09-30T00:24:15.843+0000 7f4fd907d700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:24:15.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-09-30T00:24:15.847+0000 7f0487326700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.850 INFO:tasks.ceph.osd.4.smithi146.stderr:2023-09-30T00:24:15.849+0000 7f38879b5700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T00:24:15.850 INFO:tasks.ceph.osd.6.smithi146.stderr:2023-09-30T00:24:15.849+0000 7f916a617700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T00:24:15.851 INFO:tasks.ceph.osd.5.smithi146.stderr:2023-09-30T00:24:15.849+0000 7f92d0f00700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T00:24:15.851 INFO:tasks.ceph.osd.7.smithi146.stderr:2023-09-30T00:24:15.849+0000 7f2bab7b4700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T00:24:15.855 INFO:tasks.ceph.osd.3.smithi031.stderr:2023-09-30T00:24:15.853+0000 7f736dffa700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.856 INFO:tasks.ceph.osd.7.smithi146.stderr:2023-09-30T00:24:15.855+0000 7f2ba45ab700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.856 INFO:tasks.ceph.osd.4.smithi146.stderr:2023-09-30T00:24:15.855+0000 7f38807ac700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.856 INFO:tasks.ceph.osd.6.smithi146.stderr:2023-09-30T00:24:15.855+0000 7f9161c0b700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.857 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-09-30T00:24:15.854+0000 7f4fd0671700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.857 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-09-30T00:24:15.855+0000 7f20f0f75700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:15.859 INFO:tasks.ceph.osd.5.smithi146.stderr:2023-09-30T00:24:15.858+0000 7f92c84f4700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:24:17.856 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-09-30T00:24:17.855+0000 7ffa3eeea700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:24:18.635 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:24:18.973 INFO:teuthology.misc.health.smithi031.stdout: 2023-09-30T00:24:18.973 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":15,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:24:18.869362+0000","last_up_change":"2023-09-30T00:24:16.843463+0000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:24:17.820431+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":88112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":88112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":88112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":88112}]},"public_addr":"172.21.15.31:6804/88112","cluster_addr":"172.21.15.31:6805/88112","heartbeat_back_addr":"172.21.15.31:6807/88112","heartbeat_front_addr":"172.21.15.31:6806/88112","state":["exists","up"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":88103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":88103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":88103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":88103}]},"public_addr":"172.21.15.31:6800/88103","cluster_addr":"172.21.15.31:6801/88103","heartbeat_back_addr":"172.21.15.31:6803/88103","heartbeat_front_addr":"172.21.15.31:6802/88103","state":["exists","up"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":88113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":88113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":88113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":88113}]},"public_addr":"172.21.15.31:6808/88113","cluster_addr":"172.21.15.31:6809/88113","heartbeat_back_addr":"172.21.15.31:6811/88113","heartbeat_front_addr":"172.21.15.31:6810/88113","state":["exists","up"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":88114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":88114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":88114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":88114}]},"public_addr":"172.21.15.31:6812/88114","cluster_addr":"172.21.15.31:6813/88114","heartbeat_back_addr":"172.21.15.31:6815/88114","heartbeat_front_addr":"172.21.15.31:6814/88114","state":["exists","up"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":88052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6805","nonce":88052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6807","nonce":88052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":88052}]},"public_addr":"172.21.15.146:6804/88052","cluster_addr":"172.21.15.146:6805/88052","heartbeat_back_addr":"172.21.15.146:6807/88052","heartbeat_front_addr":"172.21.15.146:6806/88052","state":["exists","up"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":88051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6801","nonce":88051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6803","nonce":88051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":88051}]},"public_addr":"172.21.15.146:6800/88051","cluster_addr":"172.21.15.146:6801/88051","heartbeat_back_addr":"172.21.15.146:6803/88051","heartbeat_front_addr":"172.21.15.146:6802/88051","state":["exists","up"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":88081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":88081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":88081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":88081}]},"public_addr":"172.21.15.146:6808/88081","cluster_addr":"172.21.15.146:6809/88081","heartbeat_back_addr":"172.21.15.146:6811/88081","heartbeat_front_addr":"172.21.15.146:6810/88081","state":["exists","up"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":88082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6813","nonce":88082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":88082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":88082}]},"public_addr":"172.21.15.146:6812/88082","cluster_addr":"172.21.15.146:6813/88082","heartbeat_back_addr":"172.21.15.146:6815/88082","heartbeat_front_addr":"172.21.15.146:6814/88082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.703696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.393167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.755648+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.685397+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.744440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.618137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.673453+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.700761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:24:18.985 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T00:24:18.986 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:24:18.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:24:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2023-09-30T00:24:19.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:24:20.892 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:24:20.904 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:24:20.905 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:24:20.905 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:24:20.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:24:20.905 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:24:21.304 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:21.316 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":87504}]},"active_addr":"172.21.15.146:6816/87504","active_change":"2023-09-30T00:24:13.753502+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3434875411}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3965408287}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":1800815157}]}]} 2023-09-30T00:24:21.321 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:24:21.321 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:24:21.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:24:21.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:21.652 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:24:20.879535+0000","last_up_change":"2023-09-30T00:24:16.843463+0000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:24:17.820431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:24:19.342552+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":88112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":88112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":88112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":88112}]},"public_addr":"172.21.15.31:6804/88112","cluster_addr":"172.21.15.31:6805/88112","heartbeat_back_addr":"172.21.15.31:6807/88112","heartbeat_front_addr":"172.21.15.31:6806/88112","state":["exists","up"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":88103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":88103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":88103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":88103}]},"public_addr":"172.21.15.31:6800/88103","cluster_addr":"172.21.15.31:6801/88103","heartbeat_back_addr":"172.21.15.31:6803/88103","heartbeat_front_addr":"172.21.15.31:6802/88103","state":["exists","up"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":88113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":88113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":88113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":88113}]},"public_addr":"172.21.15.31:6808/88113","cluster_addr":"172.21.15.31:6809/88113","heartbeat_back_addr":"172.21.15.31:6811/88113","heartbeat_front_addr":"172.21.15.31:6810/88113","state":["exists","up"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":88114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":88114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":88114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":88114}]},"public_addr":"172.21.15.31:6812/88114","cluster_addr":"172.21.15.31:6813/88114","heartbeat_back_addr":"172.21.15.31:6815/88114","heartbeat_front_addr":"172.21.15.31:6814/88114","state":["exists","up"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":88052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6805","nonce":88052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6807","nonce":88052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":88052}]},"public_addr":"172.21.15.146:6804/88052","cluster_addr":"172.21.15.146:6805/88052","heartbeat_back_addr":"172.21.15.146:6807/88052","heartbeat_front_addr":"172.21.15.146:6806/88052","state":["exists","up"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":88051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6801","nonce":88051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6803","nonce":88051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":88051}]},"public_addr":"172.21.15.146:6800/88051","cluster_addr":"172.21.15.146:6801/88051","heartbeat_back_addr":"172.21.15.146:6803/88051","heartbeat_front_addr":"172.21.15.146:6802/88051","state":["exists","up"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":88081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":88081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":88081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":88081}]},"public_addr":"172.21.15.146:6808/88081","cluster_addr":"172.21.15.146:6809/88081","heartbeat_back_addr":"172.21.15.146:6811/88081","heartbeat_front_addr":"172.21.15.146:6810/88081","state":["exists","up"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":88082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6813","nonce":88082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":88082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":88082}]},"public_addr":"172.21.15.146:6812/88082","cluster_addr":"172.21.15.146:6813/88082","heartbeat_back_addr":"172.21.15.146:6815/88082","heartbeat_front_addr":"172.21.15.146:6814/88082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.703696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.393167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.755648+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.685397+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.744440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.618137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.673453+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.700761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:24:21.666 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:24:21.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:24:21.998 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:21.998 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:24:21.885744+0000","last_up_change":"2023-09-30T00:24:16.843463+0000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:24:17.820431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:24:19.342552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":88112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":88112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":88112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":88112}]},"public_addr":"172.21.15.31:6804/88112","cluster_addr":"172.21.15.31:6805/88112","heartbeat_back_addr":"172.21.15.31:6807/88112","heartbeat_front_addr":"172.21.15.31:6806/88112","state":["exists","up"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":88103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":88103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":88103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":88103}]},"public_addr":"172.21.15.31:6800/88103","cluster_addr":"172.21.15.31:6801/88103","heartbeat_back_addr":"172.21.15.31:6803/88103","heartbeat_front_addr":"172.21.15.31:6802/88103","state":["exists","up"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":88113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":88113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":88113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":88113}]},"public_addr":"172.21.15.31:6808/88113","cluster_addr":"172.21.15.31:6809/88113","heartbeat_back_addr":"172.21.15.31:6811/88113","heartbeat_front_addr":"172.21.15.31:6810/88113","state":["exists","up"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":88114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":88114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":88114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":88114}]},"public_addr":"172.21.15.31:6812/88114","cluster_addr":"172.21.15.31:6813/88114","heartbeat_back_addr":"172.21.15.31:6815/88114","heartbeat_front_addr":"172.21.15.31:6814/88114","state":["exists","up"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":88052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6805","nonce":88052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6807","nonce":88052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":88052}]},"public_addr":"172.21.15.146:6804/88052","cluster_addr":"172.21.15.146:6805/88052","heartbeat_back_addr":"172.21.15.146:6807/88052","heartbeat_front_addr":"172.21.15.146:6806/88052","state":["exists","up"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":88051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6801","nonce":88051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6803","nonce":88051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":88051}]},"public_addr":"172.21.15.146:6800/88051","cluster_addr":"172.21.15.146:6801/88051","heartbeat_back_addr":"172.21.15.146:6803/88051","heartbeat_front_addr":"172.21.15.146:6802/88051","state":["exists","up"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":88081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":88081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":88081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":88081}]},"public_addr":"172.21.15.146:6808/88081","cluster_addr":"172.21.15.146:6809/88081","heartbeat_back_addr":"172.21.15.146:6811/88081","heartbeat_front_addr":"172.21.15.146:6810/88081","state":["exists","up"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":88082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6813","nonce":88082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":88082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":88082}]},"public_addr":"172.21.15.146:6812/88082","cluster_addr":"172.21.15.146:6813/88082","heartbeat_back_addr":"172.21.15.146:6815/88082","heartbeat_front_addr":"172.21.15.146:6814/88082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.703696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.393167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.755648+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.685397+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.744440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.618137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.673453+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:24:15.700761+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:24:22.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:24:22.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:24:22.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:24:22.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:24:22.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T00:24:22.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T00:24:22.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T00:24:22.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T00:24:22.293 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:22.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:24:22.302 INFO:teuthology.orchestra.run.smithi031.stdout:55834574850 2023-09-30T00:24:22.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:24:22.304 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:22.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:24:22.305 INFO:teuthology.orchestra.run.smithi031.stdout:55834574850 2023-09-30T00:24:22.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:24:22.308 INFO:teuthology.orchestra.run.smithi031.stdout:55834574850 2023-09-30T00:24:22.309 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:24:22.319 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:22.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:24:22.322 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:22.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:24:22.326 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:22.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:24:22.899 INFO:teuthology.orchestra.run.smithi031.stdout:55834574850 2023-09-30T00:24:22.919 INFO:teuthology.orchestra.run.smithi031.stdout:55834574849 2023-09-30T00:24:22.922 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.0 2023-09-30T00:24:22.927 INFO:teuthology.orchestra.run.smithi031.stdout:55834574850 2023-09-30T00:24:22.946 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.4 2023-09-30T00:24:22.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.7 2023-09-30T00:24:22.947 INFO:teuthology.orchestra.run.smithi031.stdout:55834574849 2023-09-30T00:24:22.958 INFO:teuthology.orchestra.run.smithi031.stdout:55834574849 2023-09-30T00:24:22.962 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.1 2023-09-30T00:24:22.978 INFO:teuthology.orchestra.run.smithi031.stdout:55834574849 2023-09-30T00:24:22.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.3 2023-09-30T00:24:22.980 INFO:teuthology.orchestra.run.smithi031.stdout:55834574849 2023-09-30T00:24:22.991 INFO:teuthology.orchestra.run.smithi031.stdout:55834574850 2023-09-30T00:24:22.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574849 for osd.5 2023-09-30T00:24:22.996 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574849 for osd.2 2023-09-30T00:24:23.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.6 2023-09-30T00:24:23.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:24:23.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:24:23.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:24:23.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:24:23.980 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:24:23.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:24:23.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:24:24.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:24:24.560 INFO:teuthology.orchestra.run.smithi031.stdout:55834574852 2023-09-30T00:24:24.573 INFO:teuthology.orchestra.run.smithi031.stdout:55834574852 2023-09-30T00:24:24.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.0 2023-09-30T00:24:24.580 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.592 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.7 2023-09-30T00:24:24.592 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.601 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:24.604 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:24.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.1 2023-09-30T00:24:24.616 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.4 2023-09-30T00:24:24.625 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.630 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:24.630 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:24.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.3 2023-09-30T00:24:24.646 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.649 INFO:teuthology.orchestra.run.smithi031.stdout:55834574852 2023-09-30T00:24:24.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2023-09-30T00:24:24.652 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.656 INFO:teuthology.orchestra.run.smithi031.stdout:55834574851 2023-09-30T00:24:24.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.6 2023-09-30T00:24:24.665 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2023-09-30T00:24:24.672 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:24.672 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:24:24.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:24:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:24:25.051 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-09-30T00:24:23.759093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":45280,"kb_used_data":2728,"kb_used_omap":0,"kb_used_meta":41216,"kb_avail":754929440,"statfs":{"total":773094113280,"available":773047746560,"internally_reserved":0,"allocated":2793472,"data_stored":1958376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42205184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:20.901461+0000","last_change":"2023-09-30T00:24:20.901461+0000","last_active":"2023-09-30T00:24:20.901461+0000","last_peered":"2023-09-30T00:24:20.901461+0000","last_clean":"2023-09-30T00:24:20.901461+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:24:20.901461+0000","last_undegraded":"2023-09-30T00:24:20.901461+0000","last_fullsized":"2023-09-30T00:24:20.901461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:41.146389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.129755+0000","last_change":"2023-09-30T00:24:21.129755+0000","last_active":"2023-09-30T00:24:21.129755+0000","last_peered":"2023-09-30T00:24:21.129755+0000","last_clean":"2023-09-30T00:24:21.129755+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:24:21.129755+0000","last_undegraded":"2023-09-30T00:24:21.129755+0000","last_fullsized":"2023-09-30T00:24:21.129755+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:48.668796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.130918+0000","last_change":"2023-09-30T00:24:21.130918+0000","last_active":"2023-09-30T00:24:21.130918+0000","last_peered":"2023-09-30T00:24:21.130918+0000","last_clean":"2023-09-30T00:24:21.130918+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:24:21.130918+0000","last_undegraded":"2023-09-30T00:24:21.130918+0000","last_fullsized":"2023-09-30T00:24:21.130918+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:34.826367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.129776+0000","last_change":"2023-09-30T00:24:21.129776+0000","last_active":"2023-09-30T00:24:21.129776+0000","last_peered":"2023-09-30T00:24:21.129776+0000","last_clean":"2023-09-30T00:24:21.129776+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:24:21.129776+0000","last_undegraded":"2023-09-30T00:24:21.129776+0000","last_fullsized":"2023-09-30T00:24:21.129776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:30.807051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.191016+0000","last_change":"2023-09-30T00:24:19.332773+0000","last_active":"2023-09-30T00:24:21.191016+0000","last_peered":"2023-09-30T00:24:21.191016+0000","last_clean":"2023-09-30T00:24:21.191016+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:24:21.191016+0000","last_undegraded":"2023-09-30T00:24:21.191016+0000","last_fullsized":"2023-09-30T00:24:21.191016+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:17.851490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:17.851490+0000","last_clean_scrub_stamp":"2023-09-30T00:24:17.851490+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:30.050419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:20.900750+0000","last_change":"2023-09-30T00:24:20.900750+0000","last_active":"2023-09-30T00:24:20.900750+0000","last_peered":"2023-09-30T00:24:20.900750+0000","last_clean":"2023-09-30T00:24:20.900750+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:24:20.900750+0000","last_undegraded":"2023-09-30T00:24:20.900750+0000","last_fullsized":"2023-09-30T00:24:20.900750+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:25.655773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.131943+0000","last_change":"2023-09-30T00:24:21.130979+0000","last_active":"2023-09-30T00:24:21.131943+0000","last_peered":"2023-09-30T00:24:21.131943+0000","last_clean":"2023-09-30T00:24:21.131943+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:24:21.131943+0000","last_undegraded":"2023-09-30T00:24:21.131943+0000","last_fullsized":"2023-09-30T00:24:21.131943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:29.405139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:20.900788+0000","last_change":"2023-09-30T00:24:20.900788+0000","last_active":"2023-09-30T00:24:20.900788+0000","last_peered":"2023-09-30T00:24:20.900788+0000","last_clean":"2023-09-30T00:24:20.900788+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:24:20.900788+0000","last_undegraded":"2023-09-30T00:24:20.900788+0000","last_fullsized":"2023-09-30T00:24:20.900788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:20.830995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.130900+0000","last_change":"2023-09-30T00:24:21.130900+0000","last_active":"2023-09-30T00:24:21.130900+0000","last_peered":"2023-09-30T00:24:21.130900+0000","last_clean":"2023-09-30T00:24:21.130900+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:24:21.130900+0000","last_undegraded":"2023-09-30T00:24:21.130900+0000","last_fullsized":"2023-09-30T00:24:21.130900+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:38.772367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6556,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365284,"statfs":{"total":96636764160,"available":96630050816,"internally_reserved":0,"allocated":679936,"data_stored":574223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.056}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6496,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365344,"statfs":{"total":96636764160,"available":96630112256,"internally_reserved":0,"allocated":679936,"data_stored":574223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6564,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365276,"statfs":{"total":96636764160,"available":96630042624,"internally_reserved":0,"allocated":688128,"data_stored":578839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.423}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1769999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:24:25.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:24:25.377 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:25.379 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:24:25.391 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-09-30T00:24:23.759093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":45280,"kb_used_data":2728,"kb_used_omap":0,"kb_used_meta":41216,"kb_avail":754929440,"statfs":{"total":773094113280,"available":773047746560,"internally_reserved":0,"allocated":2793472,"data_stored":1958376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42205184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:20.901461+0000","last_change":"2023-09-30T00:24:20.901461+0000","last_active":"2023-09-30T00:24:20.901461+0000","last_peered":"2023-09-30T00:24:20.901461+0000","last_clean":"2023-09-30T00:24:20.901461+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:24:20.901461+0000","last_undegraded":"2023-09-30T00:24:20.901461+0000","last_fullsized":"2023-09-30T00:24:20.901461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:41.146389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.129755+0000","last_change":"2023-09-30T00:24:21.129755+0000","last_active":"2023-09-30T00:24:21.129755+0000","last_peered":"2023-09-30T00:24:21.129755+0000","last_clean":"2023-09-30T00:24:21.129755+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:24:21.129755+0000","last_undegraded":"2023-09-30T00:24:21.129755+0000","last_fullsized":"2023-09-30T00:24:21.129755+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:48.668796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.130918+0000","last_change":"2023-09-30T00:24:21.130918+0000","last_active":"2023-09-30T00:24:21.130918+0000","last_peered":"2023-09-30T00:24:21.130918+0000","last_clean":"2023-09-30T00:24:21.130918+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:24:21.130918+0000","last_undegraded":"2023-09-30T00:24:21.130918+0000","last_fullsized":"2023-09-30T00:24:21.130918+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:34.826367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.129776+0000","last_change":"2023-09-30T00:24:21.129776+0000","last_active":"2023-09-30T00:24:21.129776+0000","last_peered":"2023-09-30T00:24:21.129776+0000","last_clean":"2023-09-30T00:24:21.129776+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:24:21.129776+0000","last_undegraded":"2023-09-30T00:24:21.129776+0000","last_fullsized":"2023-09-30T00:24:21.129776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:30.807051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.191016+0000","last_change":"2023-09-30T00:24:19.332773+0000","last_active":"2023-09-30T00:24:21.191016+0000","last_peered":"2023-09-30T00:24:21.191016+0000","last_clean":"2023-09-30T00:24:21.191016+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:24:21.191016+0000","last_undegraded":"2023-09-30T00:24:21.191016+0000","last_fullsized":"2023-09-30T00:24:21.191016+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:17.851490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:17.851490+0000","last_clean_scrub_stamp":"2023-09-30T00:24:17.851490+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:30.050419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:20.900750+0000","last_change":"2023-09-30T00:24:20.900750+0000","last_active":"2023-09-30T00:24:20.900750+0000","last_peered":"2023-09-30T00:24:20.900750+0000","last_clean":"2023-09-30T00:24:20.900750+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:24:20.900750+0000","last_undegraded":"2023-09-30T00:24:20.900750+0000","last_fullsized":"2023-09-30T00:24:20.900750+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:25.655773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.131943+0000","last_change":"2023-09-30T00:24:21.130979+0000","last_active":"2023-09-30T00:24:21.131943+0000","last_peered":"2023-09-30T00:24:21.131943+0000","last_clean":"2023-09-30T00:24:21.131943+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:24:21.131943+0000","last_undegraded":"2023-09-30T00:24:21.131943+0000","last_fullsized":"2023-09-30T00:24:21.131943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:29.405139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:20.900788+0000","last_change":"2023-09-30T00:24:20.900788+0000","last_active":"2023-09-30T00:24:20.900788+0000","last_peered":"2023-09-30T00:24:20.900788+0000","last_clean":"2023-09-30T00:24:20.900788+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:24:20.900788+0000","last_undegraded":"2023-09-30T00:24:20.900788+0000","last_fullsized":"2023-09-30T00:24:20.900788+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:20.830995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:24:21.130900+0000","last_change":"2023-09-30T00:24:21.130900+0000","last_active":"2023-09-30T00:24:21.130900+0000","last_peered":"2023-09-30T00:24:21.130900+0000","last_clean":"2023-09-30T00:24:21.130900+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:24:21.130900+0000","last_undegraded":"2023-09-30T00:24:21.130900+0000","last_fullsized":"2023-09-30T00:24:21.130900+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:24:19.872849+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:25:38.772367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6556,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365284,"statfs":{"total":96636764160,"available":96630050816,"internally_reserved":0,"allocated":679936,"data_stored":574223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.056}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6496,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365344,"statfs":{"total":96636764160,"available":96630112256,"internally_reserved":0,"allocated":679936,"data_stored":574223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6564,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365276,"statfs":{"total":96636764160,"available":96630042624,"internally_reserved":0,"allocated":688128,"data_stored":578839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.423}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1769999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366760,"statfs":{"total":96636764160,"available":96631562240,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":155648,"data_stored":49911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":45295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:24:25.392 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:24:25.392 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:24:25.392 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:24:25.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:25.762 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:25.763 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:28.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:29.161 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:29.161 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:32.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:32.560 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:35.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:35.955 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:35.955 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:38.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:39.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:39.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:42.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:42.754 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:42.755 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:45.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:46.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:46.153 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:49.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:49.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:49.552 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:52.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:52.947 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:52.947 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:55.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:56.343 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:56.344 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:24:59.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:24:59.741 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:24:59.741 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:02.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:03.147 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:06.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:06.543 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:09.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:09.941 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:09.942 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:12.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:13.340 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:16.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:16.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:16.743 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:19.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:20.138 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:20.138 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:23.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:23.537 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:26.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:26.937 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:26.937 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:29.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:30.338 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:30.338 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:33.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:33.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:33.739 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:36.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:37.141 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:37.141 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:40.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:40.535 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:40.536 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:43.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:43.931 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:43.931 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:46.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:47.331 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:47.331 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:50.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:50.727 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:50.727 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:53.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:54.124 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:54.124 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:25:57.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:25:57.522 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:25:57.522 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:00.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:00.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:00.917 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:03.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:04.315 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:04.315 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:07.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:07.714 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:07.714 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:10.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:11.123 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:11.124 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:14.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:14.521 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:14.521 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:17.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:17.919 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:17.919 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:20.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:21.319 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:21.319 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:24.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:24.715 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:27.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:28.111 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:31.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:31.503 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:31.504 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:34.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:34.899 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:34.900 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:37.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:38.303 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:38.303 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:41.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:41.700 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:41.700 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:44.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:45.102 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:45.103 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:48.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:48.500 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:48.500 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:51.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:51.901 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:51.901 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:54.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:55.302 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:55.303 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:26:58.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:26:58.699 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:26:58.699 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:01.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:02.099 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:02.099 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:05.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:05.496 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:08.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:08.890 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:08.890 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:11.904 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:15.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:15.680 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:15.680 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:18.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:19.077 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:19.077 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:22.091 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:22.473 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:22.474 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:25.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:25.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:25.869 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:28.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:29.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:29.268 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:32.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:32.671 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:32.671 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:35.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:36.066 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:36.066 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:39.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:39.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:39.467 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:42.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:42.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:42.865 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:45.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:46.264 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:46.265 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:49.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:49.661 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:49.661 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:52.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:53.060 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:53.060 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:56.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:56.461 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:56.461 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:27:59.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:27:59.858 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:27:59.858 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:02.873 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:03.260 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:06.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:06.656 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:09.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:10.052 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:10.052 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:13.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:13.450 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:13.450 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:16.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:16.844 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:16.845 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:19.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:20.244 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:20.244 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:23.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:23.642 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:26.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:27.044 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:27.044 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:30.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:30.442 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:33.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:33.839 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:36.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:37.241 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:37.241 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:40.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:40.642 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:40.642 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:43.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:44.043 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:44.043 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:47.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:47.443 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:47.444 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:50.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:50.843 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:50.843 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:53.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:54.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:54.240 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:28:57.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:28:57.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:28:57.637 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:00.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:01.033 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:01.033 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:04.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:04.431 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:04.431 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:07.445 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:07.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:07.826 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:10.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:11.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:11.223 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:14.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:14.621 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:14.621 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:17.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:18.020 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:18.020 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:21.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:21.416 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:21.416 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:24.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:24.812 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:27.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:29:28.212 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:28.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:29:28.226 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:29:28.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:29:28.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:28.556 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:29:18.053662+0000","last_up_change":"2023-09-30T00:24:16.843463+0000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:24:17.820431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:24:19.342552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":88112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":88112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":88112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":88112}]},"public_addr":"172.21.15.31:6804/88112","cluster_addr":"172.21.15.31:6805/88112","heartbeat_back_addr":"172.21.15.31:6807/88112","heartbeat_front_addr":"172.21.15.31:6806/88112","state":["exists","up"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":88103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":88103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":88103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":88103}]},"public_addr":"172.21.15.31:6800/88103","cluster_addr":"172.21.15.31:6801/88103","heartbeat_back_addr":"172.21.15.31:6803/88103","heartbeat_front_addr":"172.21.15.31:6802/88103","state":["exists","up"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":88113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":88113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":88113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":88113}]},"public_addr":"172.21.15.31:6808/88113","cluster_addr":"172.21.15.31:6809/88113","heartbeat_back_addr":"172.21.15.31:6811/88113","heartbeat_front_addr":"172.21.15.31:6810/88113","state":["exists","up"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":88114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":88114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":88114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":88114}]},"public_addr":"172.21.15.31:6812/88114","cluster_addr":"172.21.15.31:6813/88114","heartbeat_back_addr":"172.21.15.31:6815/88114","heartbeat_front_addr":"172.21.15.31:6814/88114","state":["exists","up"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":88052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6805","nonce":88052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6807","nonce":88052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":88052}]},"public_addr":"172.21.15.146:6804/88052","cluster_addr":"172.21.15.146:6805/88052","heartbeat_back_addr":"172.21.15.146:6807/88052","heartbeat_front_addr":"172.21.15.146:6806/88052","state":["exists","up"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":88051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6801","nonce":88051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6803","nonce":88051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":88051}]},"public_addr":"172.21.15.146:6800/88051","cluster_addr":"172.21.15.146:6801/88051","heartbeat_back_addr":"172.21.15.146:6803/88051","heartbeat_front_addr":"172.21.15.146:6802/88051","state":["exists","up"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":88081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":88081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":88081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":88081}]},"public_addr":"172.21.15.146:6808/88081","cluster_addr":"172.21.15.146:6809/88081","heartbeat_back_addr":"172.21.15.146:6811/88081","heartbeat_front_addr":"172.21.15.146:6810/88081","state":["exists","up"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":88082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6813","nonce":88082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":88082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":88082}]},"public_addr":"172.21.15.146:6812/88082","cluster_addr":"172.21.15.146:6813/88082","heartbeat_back_addr":"172.21.15.146:6815/88082","heartbeat_front_addr":"172.21.15.146:6814/88082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:29:00.951191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:59.454171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:42.329368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:58.015825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:29:17.764025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:13.464969+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:14.332961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:29:11.581585+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:29:28.570 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:29:28.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:29:28.894 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:29:28.907 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T00:29:27.826737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":48088,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":754926632,"statfs":{"total":773094113280,"available":773044871168,"internally_reserved":0,"allocated":4677632,"data_stored":2956344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002119"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.048852+0000","last_change":"2023-09-30T00:28:18.718832+0000","last_active":"2023-09-30T00:29:12.048852+0000","last_peered":"2023-09-30T00:29:12.048852+0000","last_clean":"2023-09-30T00:29:12.048852+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:29:12.048852+0000","last_undegraded":"2023-09-30T00:29:12.048852+0000","last_fullsized":"2023-09-30T00:29:12.048852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:18.718735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:18.718735+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:28.151748+0000","scrub_duration":0.001786316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.049134+0000","last_change":"2023-09-30T00:28:23.947821+0000","last_active":"2023-09-30T00:29:12.049134+0000","last_peered":"2023-09-30T00:29:12.049134+0000","last_clean":"2023-09-30T00:29:12.049134+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:29:12.049134+0000","last_undegraded":"2023-09-30T00:29:12.049134+0000","last_fullsized":"2023-09-30T00:29:12.049134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:23.947740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:23.947740+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:27.863817+0000","scrub_duration":0.002233348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944502+0000","last_change":"2023-09-30T00:28:21.511933+0000","last_active":"2023-09-30T00:29:12.944502+0000","last_peered":"2023-09-30T00:29:12.944502+0000","last_clean":"2023-09-30T00:29:12.944502+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:29:12.944502+0000","last_undegraded":"2023-09-30T00:29:12.944502+0000","last_fullsized":"2023-09-30T00:29:12.944502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:21.511819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:21.511819+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:41.914937+0000","scrub_duration":0.0016831120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944516+0000","last_change":"2023-09-30T00:28:17.446760+0000","last_active":"2023-09-30T00:29:12.944516+0000","last_peered":"2023-09-30T00:29:12.944516+0000","last_clean":"2023-09-30T00:29:12.944516+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:29:12.944516+0000","last_undegraded":"2023-09-30T00:29:12.944516+0000","last_fullsized":"2023-09-30T00:29:12.944516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:17.446653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:17.446653+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:39.685841+0000","scrub_duration":0.0021246559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":232,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:20.841814+0000","last_change":"2023-09-30T00:29:20.841771+0000","last_active":"2023-09-30T00:29:20.841814+0000","last_peered":"2023-09-30T00:29:20.841814+0000","last_clean":"2023-09-30T00:29:20.841814+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:29:20.841814+0000","last_undegraded":"2023-09-30T00:29:20.841814+0000","last_fullsized":"2023-09-30T00:29:20.841814+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2023-09-30T00:29:20.841644+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2023-09-30T00:25:30.067930+0000","last_clean_scrub_stamp":"2023-09-30T00:29:20.841644+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:48.971707+0000","scrub_duration":0.0023337359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:21.482397+0000","last_change":"2023-09-30T00:29:21.482365+0000","last_active":"2023-09-30T00:29:21.482397+0000","last_peered":"2023-09-30T00:29:21.482397+0000","last_clean":"2023-09-30T00:29:21.482397+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:29:21.482397+0000","last_undegraded":"2023-09-30T00:29:21.482397+0000","last_fullsized":"2023-09-30T00:29:21.482397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:21.482266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:29:21.482266+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:40.264629+0000","scrub_duration":0.001809995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:25.265414+0000","last_change":"2023-09-30T00:29:25.265364+0000","last_active":"2023-09-30T00:29:25.265414+0000","last_peered":"2023-09-30T00:29:25.265414+0000","last_clean":"2023-09-30T00:29:25.265414+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:29:25.265414+0000","last_undegraded":"2023-09-30T00:29:25.265414+0000","last_fullsized":"2023-09-30T00:29:25.265414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:25.265274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:28:02.446802+0000","last_clean_scrub_stamp":"2023-09-30T00:29:25.265274+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:52.997972+0000","scrub_duration":0.0016202670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.288971+0000","last_change":"2023-09-30T00:28:46.895512+0000","last_active":"2023-09-30T00:29:12.288971+0000","last_peered":"2023-09-30T00:29:12.288971+0000","last_clean":"2023-09-30T00:29:12.288971+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:29:12.288971+0000","last_undegraded":"2023-09-30T00:29:12.288971+0000","last_fullsized":"2023-09-30T00:29:12.288971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:46.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:26:28.175176+0000","last_clean_scrub_stamp":"2023-09-30T00:28:46.895402+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:59.201765+0000","scrub_duration":0.0018973250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.049134+0000","last_change":"2023-09-30T00:27:58.265696+0000","last_active":"2023-09-30T00:29:12.049134+0000","last_peered":"2023-09-30T00:29:12.049134+0000","last_clean":"2023-09-30T00:29:12.049134+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:29:12.049134+0000","last_undegraded":"2023-09-30T00:29:12.049134+0000","last_fullsized":"2023-09-30T00:29:12.049134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:27:58.265580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:27:58.265580+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:27.774996+0000","scrub_duration":0.0021068150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":5.4749999999999996,"15min":2.1709999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81100000000000005,"5min":518.15700000000004,"15min":518.15700000000004},"last":0.496},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.95399999999999996,"15min":1.3340000000000001},"min":{"1min":0.318,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":20.98,"15min":20.98},"last":0.67400000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":36.191000000000003,"5min":7.7039999999999997,"15min":2.944},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":499.12700000000001,"5min":499.12700000000001,"15min":499.12700000000001},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":5.8799999999999999,"15min":2.3740000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":553.47799999999995,"15min":553.47799999999995},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":17.440000000000001,"15min":18.201000000000001},"min":{"1min":0.432,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78800000000000003,"5min":772.48099999999999,"15min":772.48099999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":47.009,"5min":9.9049999999999994,"15min":3.722},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":742.43200000000002,"5min":742.43200000000002,"15min":742.43200000000002},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":2.431,"15min":1.248},"min":{"1min":0.375,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76500000000000001,"5min":197.946,"15min":197.946},"last":0.42499999999999999},{"interface":"front","average":{"1min":53.822000000000003,"5min":11.23,"15min":4.1870000000000003},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":852.97199999999998,"5min":852.97199999999998,"15min":852.97199999999998},"last":0.435}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.59499999999999997},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83399999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.374}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":51.947000000000003,"5min":11.907,"15min":4.375},"min":{"1min":0.50800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":821.89999999999998,"5min":821.89999999999998,"15min":821.89999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84499999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.40999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6600,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365240,"statfs":{"total":96636764160,"available":96630005760,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":39.131999999999998,"5min":8.3089999999999993,"15min":3.2120000000000002},"min":{"1min":0.498,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":770.97900000000004,"5min":770.97900000000004,"15min":770.97900000000004},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":8.7059999999999995,"15min":16.792000000000002},"min":{"1min":0.47099999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":486.19,"15min":486.19},"last":909.92600000000004},{"interface":"front","average":{"1min":37.652000000000001,"5min":7.9870000000000001,"15min":3.0609999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":742.36199999999997,"5min":742.36199999999997,"15min":742.36199999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":52.683,"5min":24.893000000000001,"15min":8.7520000000000007},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":990.30100000000004,"5min":990.30100000000004,"15min":990.30100000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.63200000000000001},"min":{"1min":0.46600000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88500000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":16.731999999999999,"5min":3.8199999999999998,"15min":1.673},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":323.27600000000001,"5min":323.27600000000001,"15min":323.27600000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.65100000000000002,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69799999999999995,"5min":11.202,"15min":11.202},"last":0.379}]},{"osd":5,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":10.314,"15min":3.8260000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":877.19000000000005,"15min":877.19000000000005},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":30.800999999999998,"5min":6.6070000000000002,"15min":2.5840000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":606.13499999999999,"5min":606.13499999999999,"15min":606.13499999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":37.212000000000003,"5min":7.9219999999999997,"15min":3.052},"min":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":733.37400000000002,"5min":733.37400000000002,"15min":733.37400000000002},"last":0.438}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":27.849,"5min":12.656000000000001,"15min":4.4989999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":499.25999999999999,"5min":743.42399999999998,"15min":743.42399999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.371,"5min":9.3840000000000003,"15min":3.4180000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61199999999999999,"5min":845.18100000000004,"15min":845.18100000000004},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":17.248999999999999,"15min":19.265000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":668.12900000000002,"5min":668.12900000000002,"15min":668.12900000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":10.191000000000001,"15min":19.959},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57499999999999996,"5min":585.68600000000004,"15min":585.68600000000004},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.9780000000000002,"15min":5.5190000000000001},"min":{"1min":0.28000000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61499999999999999,"5min":148.685,"15min":148.685},"last":0.58099999999999996},{"interface":"front","average":{"1min":27.045000000000002,"5min":5.835,"15min":2.3260000000000001},"min":{"1min":0.318,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":532.24199999999996,"5min":532.24199999999996,"15min":532.24199999999996},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":15.055999999999999,"15min":29.178000000000001},"min":{"1min":0.45900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84599999999999997,"5min":501.096,"15min":501.096},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":8.9550000000000001,"15min":17.300999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.79400000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":16.785,"15min":18.381},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.82899999999999996,"5min":962.88,"15min":962.88},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":9.0150000000000006,"15min":3.4870000000000001},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":794.64200000000005,"15min":794.64200000000005},"last":0.89200000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":14.795,"15min":28.966999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77000000000000002,"5min":813.57399999999996,"15min":813.57399999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67600000000000005,"15min":0.73399999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.90400000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.44,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71999999999999997,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.66600000000000004,"15min":0.69199999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78900000000000003,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42999999999999999,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.497,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.42599999999999999,"15min":0.41999999999999998},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60599999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":41.329000000000001,"5min":8.8130000000000006,"15min":3.2309999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":655.24400000000003,"5min":655.24400000000003,"15min":655.24400000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":12.734,"5min":3.0049999999999999,"15min":1.4830000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":198.53800000000001,"5min":198.53800000000001,"15min":198.53800000000001},"last":0.432},{"interface":"front","average":{"1min":0.434,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":5.5990000000000002,"15min":10.590999999999999},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68200000000000005,"5min":275.43200000000002,"15min":275.43200000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.70099999999999996,"15min":0.78200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77600000000000002,"5min":5.1269999999999998,"15min":5.1269999999999998},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":48.792000000000002,"5min":10.356999999999999,"15min":4},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":772.57299999999998,"5min":772.57299999999998,"15min":772.57299999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56100000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.64700000000000002,"15min":0.66000000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.70799999999999996,"5min":1.046,"15min":1.046},"last":0.501}]},{"osd":7,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":14.56,"15min":5.2869999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73199999999999998,"5min":990.16700000000003,"15min":990.16700000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":2.6280000000000001,"15min":1.45},"min":{"1min":0.46600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.75900000000000001,"5min":198.173,"15min":198.173},"last":0.627}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":29.780000000000001,"5min":18.908000000000001,"15min":24.202999999999999},"min":{"1min":0.34000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":499.13,"5min":586.52599999999995,"15min":586.52599999999995},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.59799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63600000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":9.2240000000000002,"15min":3.4119999999999999},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70199999999999996,"5min":655.45500000000004,"15min":655.45500000000004},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":8.4969999999999999,"15min":3.2869999999999999},"min":{"1min":0.49199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":708.62400000000002,"15min":708.62400000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":7.3339999999999996,"15min":25.207000000000001},"min":{"1min":0.46800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89500000000000002,"5min":536.221,"15min":536.221},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":20.416,"5min":13.345000000000001,"15min":4.9029999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":356.43000000000001,"5min":744.32600000000002,"15min":744.32600000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":9.8510000000000009,"15min":3.75},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90200000000000002,"5min":783.25300000000004,"15min":783.25300000000004},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":83.608999999999995,"5min":17.245999999999999,"15min":6.2199999999999998},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":678.30799999999999,"5min":678.30799999999999,"15min":678.30799999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.68799999999999994},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.60399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.64000000000000001,"15min":0.84799999999999998},"min":{"1min":0.28899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.433},{"interface":"front","average":{"1min":0.39800000000000002,"5min":18.452999999999999,"15min":35.728000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":733.85000000000002,"15min":733.85000000000002},"last":0.42199999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":15.871,"5min":15.871,"15min":15.871},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.84199999999999997,"15min":1.264},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53700000000000003,"5min":9.157,"15min":9.157},"last":986.77999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.68100000000000005,"15min":0.93700000000000006},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60399999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.54700000000000004,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57799999999999996,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":4.1269999999999998,"15min":2.1869999999999998},"min":{"1min":0.46500000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.83899999999999997,"5min":277.21499999999997,"15min":277.21499999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":16.873999999999999,"5min":3.9790000000000001,"15min":1.8899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":309.69099999999997,"5min":309.69099999999997,"15min":309.69099999999997},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":38.488999999999997,"5min":8.4239999999999995,"15min":3.581},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":668.34500000000003,"5min":668.34500000000003,"15min":668.34500000000003},"last":444.73000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.89000000000000001,"15min":1.1299999999999999},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86499999999999999,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.80000000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.92500000000000004,"5min":3.5289999999999999,"15min":3.5289999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.86699999999999999,"15min":1.091},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.879,"5min":9.0950000000000006,"15min":9.0950000000000006},"last":0.60199999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.77400000000000002,"15min":0.88400000000000001},"min":{"1min":0.433,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90100000000000002,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.85499999999999998,"15min":0.871},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79800000000000004,"5min":11.789999999999999,"15min":11.789999999999999},"last":0.65800000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":7.1230000000000002,"15min":13.659000000000001},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":360.43599999999998,"15min":360.43599999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.83199999999999996,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":11.308999999999999,"15min":4.1859999999999999},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81100000000000005,"5min":909.94100000000003,"15min":909.94100000000003},"last":0.77300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63400000000000001,"15min":0.64500000000000002},"min":{"1min":0.48299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80100000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.749},{"interface":"front","average":{"1min":0.58799999999999997,"5min":12.667,"15min":4.6589999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":963.85500000000002,"15min":963.85500000000002},"last":0.66200000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.8029999999999999,"15min":6.9450000000000003},"min":{"1min":0.53400000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94599999999999995,"5min":182.018,"15min":182.018},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":7.734,"15min":3.024},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90900000000000003,"5min":565.77200000000005,"15min":565.77200000000005},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":21.582999999999998,"5min":4.7290000000000001,"15min":1.917},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":463.30700000000002,"5min":463.30700000000002,"15min":463.30700000000002},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":6.194,"15min":2.4159999999999999},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83999999999999997,"5min":453.32400000000001,"15min":453.32400000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":13.493,"15min":4.8789999999999996},"min":{"1min":0.40799999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89000000000000001,"5min":596.84500000000003,"15min":596.84500000000003},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":10.733000000000001,"15min":20.893999999999998},"min":{"1min":0.40100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92300000000000004,"5min":585.08000000000004,"15min":585.08000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":36.701999999999998,"5min":21.587,"15min":30.317},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.38},"max":{"1min":794.49000000000001,"5min":794.49000000000001,"15min":794.49000000000001},"last":0.80900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":17.228999999999999,"15min":6.1539999999999999},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76600000000000001,"5min":954.57299999999998,"15min":954.57299999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":5.3689999999999998,"15min":2.1989999999999998},"min":{"1min":0.437,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82899999999999996,"5min":453.76100000000002,"15min":453.76100000000002},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":3.3879999999999999,"5min":23.856999999999999,"15min":40.771000000000001},"min":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":51.552999999999997,"5min":973.71299999999997,"15min":973.71299999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":8.8759999999999994,"15min":3.3639999999999999},"min":{"1min":0.38700000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71599999999999997,"5min":788.19899999999996,"15min":788.19899999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70099999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":12.221,"5min":2.9159999999999999,"15min":1.383},"min":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":198.43000000000001,"5min":198.43000000000001,"15min":198.43000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.628,"15min":0.65100000000000002},"min":{"1min":0.51200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69299999999999995,"5min":1.292,"15min":1.292},"last":0.623}]},{"osd":4,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48099999999999998,"15min":0.497},"min":{"1min":0.35499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72799999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":15.48,"15min":30.431999999999999},"min":{"1min":0.441,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76400000000000001,"5min":784.48299999999995,"15min":784.48299999999995},"last":0.42599999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":43.857999999999997,"5min":25.468,"15min":35.968000000000004},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":736.56700000000001,"5min":853.28200000000004,"15min":853.28200000000004},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:29:28.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:29:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:29.245 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:29:29.257 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T00:29:27.826737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":48088,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":754926632,"statfs":{"total":773094113280,"available":773044871168,"internally_reserved":0,"allocated":4677632,"data_stored":2956344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002119"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.048852+0000","last_change":"2023-09-30T00:28:18.718832+0000","last_active":"2023-09-30T00:29:12.048852+0000","last_peered":"2023-09-30T00:29:12.048852+0000","last_clean":"2023-09-30T00:29:12.048852+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:29:12.048852+0000","last_undegraded":"2023-09-30T00:29:12.048852+0000","last_fullsized":"2023-09-30T00:29:12.048852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:18.718735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:18.718735+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:28.151748+0000","scrub_duration":0.001786316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.049134+0000","last_change":"2023-09-30T00:28:23.947821+0000","last_active":"2023-09-30T00:29:12.049134+0000","last_peered":"2023-09-30T00:29:12.049134+0000","last_clean":"2023-09-30T00:29:12.049134+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:29:12.049134+0000","last_undegraded":"2023-09-30T00:29:12.049134+0000","last_fullsized":"2023-09-30T00:29:12.049134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:23.947740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:23.947740+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:27.863817+0000","scrub_duration":0.002233348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944502+0000","last_change":"2023-09-30T00:28:21.511933+0000","last_active":"2023-09-30T00:29:12.944502+0000","last_peered":"2023-09-30T00:29:12.944502+0000","last_clean":"2023-09-30T00:29:12.944502+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:29:12.944502+0000","last_undegraded":"2023-09-30T00:29:12.944502+0000","last_fullsized":"2023-09-30T00:29:12.944502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:21.511819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:21.511819+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:41.914937+0000","scrub_duration":0.0016831120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944516+0000","last_change":"2023-09-30T00:28:17.446760+0000","last_active":"2023-09-30T00:29:12.944516+0000","last_peered":"2023-09-30T00:29:12.944516+0000","last_clean":"2023-09-30T00:29:12.944516+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:29:12.944516+0000","last_undegraded":"2023-09-30T00:29:12.944516+0000","last_fullsized":"2023-09-30T00:29:12.944516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:17.446653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:17.446653+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:39.685841+0000","scrub_duration":0.0021246559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":232,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:20.841814+0000","last_change":"2023-09-30T00:29:20.841771+0000","last_active":"2023-09-30T00:29:20.841814+0000","last_peered":"2023-09-30T00:29:20.841814+0000","last_clean":"2023-09-30T00:29:20.841814+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:29:20.841814+0000","last_undegraded":"2023-09-30T00:29:20.841814+0000","last_fullsized":"2023-09-30T00:29:20.841814+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2023-09-30T00:29:20.841644+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2023-09-30T00:25:30.067930+0000","last_clean_scrub_stamp":"2023-09-30T00:29:20.841644+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:48.971707+0000","scrub_duration":0.0023337359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:21.482397+0000","last_change":"2023-09-30T00:29:21.482365+0000","last_active":"2023-09-30T00:29:21.482397+0000","last_peered":"2023-09-30T00:29:21.482397+0000","last_clean":"2023-09-30T00:29:21.482397+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:29:21.482397+0000","last_undegraded":"2023-09-30T00:29:21.482397+0000","last_fullsized":"2023-09-30T00:29:21.482397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:21.482266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:29:21.482266+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:40.264629+0000","scrub_duration":0.001809995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:25.265414+0000","last_change":"2023-09-30T00:29:25.265364+0000","last_active":"2023-09-30T00:29:25.265414+0000","last_peered":"2023-09-30T00:29:25.265414+0000","last_clean":"2023-09-30T00:29:25.265414+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:29:25.265414+0000","last_undegraded":"2023-09-30T00:29:25.265414+0000","last_fullsized":"2023-09-30T00:29:25.265414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:25.265274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:28:02.446802+0000","last_clean_scrub_stamp":"2023-09-30T00:29:25.265274+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:52.997972+0000","scrub_duration":0.0016202670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.288971+0000","last_change":"2023-09-30T00:28:46.895512+0000","last_active":"2023-09-30T00:29:12.288971+0000","last_peered":"2023-09-30T00:29:12.288971+0000","last_clean":"2023-09-30T00:29:12.288971+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:29:12.288971+0000","last_undegraded":"2023-09-30T00:29:12.288971+0000","last_fullsized":"2023-09-30T00:29:12.288971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:46.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:26:28.175176+0000","last_clean_scrub_stamp":"2023-09-30T00:28:46.895402+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:59.201765+0000","scrub_duration":0.0018973250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.049134+0000","last_change":"2023-09-30T00:27:58.265696+0000","last_active":"2023-09-30T00:29:12.049134+0000","last_peered":"2023-09-30T00:29:12.049134+0000","last_clean":"2023-09-30T00:29:12.049134+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:29:12.049134+0000","last_undegraded":"2023-09-30T00:29:12.049134+0000","last_fullsized":"2023-09-30T00:29:12.049134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:27:58.265580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:27:58.265580+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:27.774996+0000","scrub_duration":0.0021068150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":5.4749999999999996,"15min":2.1709999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81100000000000005,"5min":518.15700000000004,"15min":518.15700000000004},"last":0.496},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.95399999999999996,"15min":1.3340000000000001},"min":{"1min":0.318,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":20.98,"15min":20.98},"last":0.67400000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":36.191000000000003,"5min":7.7039999999999997,"15min":2.944},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":499.12700000000001,"5min":499.12700000000001,"15min":499.12700000000001},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":5.8799999999999999,"15min":2.3740000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":553.47799999999995,"15min":553.47799999999995},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":17.440000000000001,"15min":18.201000000000001},"min":{"1min":0.432,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78800000000000003,"5min":772.48099999999999,"15min":772.48099999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":47.009,"5min":9.9049999999999994,"15min":3.722},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":742.43200000000002,"5min":742.43200000000002,"15min":742.43200000000002},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":2.431,"15min":1.248},"min":{"1min":0.375,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76500000000000001,"5min":197.946,"15min":197.946},"last":0.42499999999999999},{"interface":"front","average":{"1min":53.822000000000003,"5min":11.23,"15min":4.1870000000000003},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":852.97199999999998,"5min":852.97199999999998,"15min":852.97199999999998},"last":0.435}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.59499999999999997},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83399999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.374}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":51.947000000000003,"5min":11.907,"15min":4.375},"min":{"1min":0.50800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":821.89999999999998,"5min":821.89999999999998,"15min":821.89999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84499999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.40999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6600,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365240,"statfs":{"total":96636764160,"available":96630005760,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":39.131999999999998,"5min":8.3089999999999993,"15min":3.2120000000000002},"min":{"1min":0.498,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":770.97900000000004,"5min":770.97900000000004,"15min":770.97900000000004},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":8.7059999999999995,"15min":16.792000000000002},"min":{"1min":0.47099999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":486.19,"15min":486.19},"last":909.92600000000004},{"interface":"front","average":{"1min":37.652000000000001,"5min":7.9870000000000001,"15min":3.0609999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":742.36199999999997,"5min":742.36199999999997,"15min":742.36199999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":52.683,"5min":24.893000000000001,"15min":8.7520000000000007},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":990.30100000000004,"5min":990.30100000000004,"15min":990.30100000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.63200000000000001},"min":{"1min":0.46600000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88500000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":16.731999999999999,"5min":3.8199999999999998,"15min":1.673},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":323.27600000000001,"5min":323.27600000000001,"15min":323.27600000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.65100000000000002,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69799999999999995,"5min":11.202,"15min":11.202},"last":0.379}]},{"osd":5,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":10.314,"15min":3.8260000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":877.19000000000005,"15min":877.19000000000005},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":30.800999999999998,"5min":6.6070000000000002,"15min":2.5840000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":606.13499999999999,"5min":606.13499999999999,"15min":606.13499999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":37.212000000000003,"5min":7.9219999999999997,"15min":3.052},"min":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":733.37400000000002,"5min":733.37400000000002,"15min":733.37400000000002},"last":0.438}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":27.849,"5min":12.656000000000001,"15min":4.4989999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":499.25999999999999,"5min":743.42399999999998,"15min":743.42399999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.371,"5min":9.3840000000000003,"15min":3.4180000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61199999999999999,"5min":845.18100000000004,"15min":845.18100000000004},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":17.248999999999999,"15min":19.265000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":668.12900000000002,"5min":668.12900000000002,"15min":668.12900000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":10.191000000000001,"15min":19.959},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57499999999999996,"5min":585.68600000000004,"15min":585.68600000000004},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.9780000000000002,"15min":5.5190000000000001},"min":{"1min":0.28000000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61499999999999999,"5min":148.685,"15min":148.685},"last":0.58099999999999996},{"interface":"front","average":{"1min":27.045000000000002,"5min":5.835,"15min":2.3260000000000001},"min":{"1min":0.318,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":532.24199999999996,"5min":532.24199999999996,"15min":532.24199999999996},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":15.055999999999999,"15min":29.178000000000001},"min":{"1min":0.45900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84599999999999997,"5min":501.096,"15min":501.096},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":8.9550000000000001,"15min":17.300999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.79400000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":16.785,"15min":18.381},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.82899999999999996,"5min":962.88,"15min":962.88},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":9.0150000000000006,"15min":3.4870000000000001},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":794.64200000000005,"15min":794.64200000000005},"last":0.89200000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":14.795,"15min":28.966999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77000000000000002,"5min":813.57399999999996,"15min":813.57399999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67600000000000005,"15min":0.73399999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.90400000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.44,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71999999999999997,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.66600000000000004,"15min":0.69199999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78900000000000003,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42999999999999999,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.497,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.42599999999999999,"15min":0.41999999999999998},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60599999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":41.329000000000001,"5min":8.8130000000000006,"15min":3.2309999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":655.24400000000003,"5min":655.24400000000003,"15min":655.24400000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":12.734,"5min":3.0049999999999999,"15min":1.4830000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":198.53800000000001,"5min":198.53800000000001,"15min":198.53800000000001},"last":0.432},{"interface":"front","average":{"1min":0.434,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":5.5990000000000002,"15min":10.590999999999999},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68200000000000005,"5min":275.43200000000002,"15min":275.43200000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.70099999999999996,"15min":0.78200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77600000000000002,"5min":5.1269999999999998,"15min":5.1269999999999998},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":48.792000000000002,"5min":10.356999999999999,"15min":4},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":772.57299999999998,"5min":772.57299999999998,"15min":772.57299999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56100000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.64700000000000002,"15min":0.66000000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.70799999999999996,"5min":1.046,"15min":1.046},"last":0.501}]},{"osd":7,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":14.56,"15min":5.2869999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73199999999999998,"5min":990.16700000000003,"15min":990.16700000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":2.6280000000000001,"15min":1.45},"min":{"1min":0.46600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.75900000000000001,"5min":198.173,"15min":198.173},"last":0.627}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":29.780000000000001,"5min":18.908000000000001,"15min":24.202999999999999},"min":{"1min":0.34000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":499.13,"5min":586.52599999999995,"15min":586.52599999999995},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.59799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63600000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":9.2240000000000002,"15min":3.4119999999999999},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70199999999999996,"5min":655.45500000000004,"15min":655.45500000000004},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":8.4969999999999999,"15min":3.2869999999999999},"min":{"1min":0.49199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":708.62400000000002,"15min":708.62400000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":7.3339999999999996,"15min":25.207000000000001},"min":{"1min":0.46800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89500000000000002,"5min":536.221,"15min":536.221},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":20.416,"5min":13.345000000000001,"15min":4.9029999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":356.43000000000001,"5min":744.32600000000002,"15min":744.32600000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":9.8510000000000009,"15min":3.75},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90200000000000002,"5min":783.25300000000004,"15min":783.25300000000004},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":83.608999999999995,"5min":17.245999999999999,"15min":6.2199999999999998},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":678.30799999999999,"5min":678.30799999999999,"15min":678.30799999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.68799999999999994},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.60399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.64000000000000001,"15min":0.84799999999999998},"min":{"1min":0.28899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.433},{"interface":"front","average":{"1min":0.39800000000000002,"5min":18.452999999999999,"15min":35.728000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":733.85000000000002,"15min":733.85000000000002},"last":0.42199999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":15.871,"5min":15.871,"15min":15.871},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.84199999999999997,"15min":1.264},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53700000000000003,"5min":9.157,"15min":9.157},"last":986.77999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.68100000000000005,"15min":0.93700000000000006},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60399999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.54700000000000004,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57799999999999996,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":4.1269999999999998,"15min":2.1869999999999998},"min":{"1min":0.46500000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.83899999999999997,"5min":277.21499999999997,"15min":277.21499999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":16.873999999999999,"5min":3.9790000000000001,"15min":1.8899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":309.69099999999997,"5min":309.69099999999997,"15min":309.69099999999997},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":38.488999999999997,"5min":8.4239999999999995,"15min":3.581},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":668.34500000000003,"5min":668.34500000000003,"15min":668.34500000000003},"last":444.73000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.89000000000000001,"15min":1.1299999999999999},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86499999999999999,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.80000000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.92500000000000004,"5min":3.5289999999999999,"15min":3.5289999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.86699999999999999,"15min":1.091},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.879,"5min":9.0950000000000006,"15min":9.0950000000000006},"last":0.60199999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.77400000000000002,"15min":0.88400000000000001},"min":{"1min":0.433,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90100000000000002,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.85499999999999998,"15min":0.871},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79800000000000004,"5min":11.789999999999999,"15min":11.789999999999999},"last":0.65800000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":7.1230000000000002,"15min":13.659000000000001},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":360.43599999999998,"15min":360.43599999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.83199999999999996,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":11.308999999999999,"15min":4.1859999999999999},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81100000000000005,"5min":909.94100000000003,"15min":909.94100000000003},"last":0.77300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63400000000000001,"15min":0.64500000000000002},"min":{"1min":0.48299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80100000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.749},{"interface":"front","average":{"1min":0.58799999999999997,"5min":12.667,"15min":4.6589999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":963.85500000000002,"15min":963.85500000000002},"last":0.66200000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.8029999999999999,"15min":6.9450000000000003},"min":{"1min":0.53400000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94599999999999995,"5min":182.018,"15min":182.018},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":7.734,"15min":3.024},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90900000000000003,"5min":565.77200000000005,"15min":565.77200000000005},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":21.582999999999998,"5min":4.7290000000000001,"15min":1.917},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":463.30700000000002,"5min":463.30700000000002,"15min":463.30700000000002},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":6.194,"15min":2.4159999999999999},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83999999999999997,"5min":453.32400000000001,"15min":453.32400000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":13.493,"15min":4.8789999999999996},"min":{"1min":0.40799999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89000000000000001,"5min":596.84500000000003,"15min":596.84500000000003},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":10.733000000000001,"15min":20.893999999999998},"min":{"1min":0.40100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92300000000000004,"5min":585.08000000000004,"15min":585.08000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":36.701999999999998,"5min":21.587,"15min":30.317},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.38},"max":{"1min":794.49000000000001,"5min":794.49000000000001,"15min":794.49000000000001},"last":0.80900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":17.228999999999999,"15min":6.1539999999999999},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76600000000000001,"5min":954.57299999999998,"15min":954.57299999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":5.3689999999999998,"15min":2.1989999999999998},"min":{"1min":0.437,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82899999999999996,"5min":453.76100000000002,"15min":453.76100000000002},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":3.3879999999999999,"5min":23.856999999999999,"15min":40.771000000000001},"min":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":51.552999999999997,"5min":973.71299999999997,"15min":973.71299999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":8.8759999999999994,"15min":3.3639999999999999},"min":{"1min":0.38700000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71599999999999997,"5min":788.19899999999996,"15min":788.19899999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70099999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":12.221,"5min":2.9159999999999999,"15min":1.383},"min":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":198.43000000000001,"5min":198.43000000000001,"15min":198.43000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.628,"15min":0.65100000000000002},"min":{"1min":0.51200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69299999999999995,"5min":1.292,"15min":1.292},"last":0.623}]},{"osd":4,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48099999999999998,"15min":0.497},"min":{"1min":0.35499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72799999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":15.48,"15min":30.431999999999999},"min":{"1min":0.441,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76400000000000001,"5min":784.48299999999995,"15min":784.48299999999995},"last":0.42599999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":43.857999999999997,"5min":25.468,"15min":35.968000000000004},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":736.56700000000001,"5min":853.28200000000004,"15min":853.28200000000004},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:29:29.259 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:29:29.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:29:29.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:29:29.598 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T00:29:27.826737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":48088,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":754926632,"statfs":{"total":773094113280,"available":773044871168,"internally_reserved":0,"allocated":4677632,"data_stored":2956344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002119"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.048852+0000","last_change":"2023-09-30T00:28:18.718832+0000","last_active":"2023-09-30T00:29:12.048852+0000","last_peered":"2023-09-30T00:29:12.048852+0000","last_clean":"2023-09-30T00:29:12.048852+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:29:12.048852+0000","last_undegraded":"2023-09-30T00:29:12.048852+0000","last_fullsized":"2023-09-30T00:29:12.048852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:18.718735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:18.718735+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:28.151748+0000","scrub_duration":0.001786316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.049134+0000","last_change":"2023-09-30T00:28:23.947821+0000","last_active":"2023-09-30T00:29:12.049134+0000","last_peered":"2023-09-30T00:29:12.049134+0000","last_clean":"2023-09-30T00:29:12.049134+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:29:12.049134+0000","last_undegraded":"2023-09-30T00:29:12.049134+0000","last_fullsized":"2023-09-30T00:29:12.049134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:23.947740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:23.947740+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:27.863817+0000","scrub_duration":0.002233348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944502+0000","last_change":"2023-09-30T00:28:21.511933+0000","last_active":"2023-09-30T00:29:12.944502+0000","last_peered":"2023-09-30T00:29:12.944502+0000","last_clean":"2023-09-30T00:29:12.944502+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:29:12.944502+0000","last_undegraded":"2023-09-30T00:29:12.944502+0000","last_fullsized":"2023-09-30T00:29:12.944502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:21.511819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:21.511819+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:41.914937+0000","scrub_duration":0.0016831120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944516+0000","last_change":"2023-09-30T00:28:17.446760+0000","last_active":"2023-09-30T00:29:12.944516+0000","last_peered":"2023-09-30T00:29:12.944516+0000","last_clean":"2023-09-30T00:29:12.944516+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:29:12.944516+0000","last_undegraded":"2023-09-30T00:29:12.944516+0000","last_fullsized":"2023-09-30T00:29:12.944516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:17.446653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:17.446653+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:39.685841+0000","scrub_duration":0.0021246559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":232,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:20.841814+0000","last_change":"2023-09-30T00:29:20.841771+0000","last_active":"2023-09-30T00:29:20.841814+0000","last_peered":"2023-09-30T00:29:20.841814+0000","last_clean":"2023-09-30T00:29:20.841814+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:29:20.841814+0000","last_undegraded":"2023-09-30T00:29:20.841814+0000","last_fullsized":"2023-09-30T00:29:20.841814+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2023-09-30T00:29:20.841644+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2023-09-30T00:25:30.067930+0000","last_clean_scrub_stamp":"2023-09-30T00:29:20.841644+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:48.971707+0000","scrub_duration":0.0023337359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:21.482397+0000","last_change":"2023-09-30T00:29:21.482365+0000","last_active":"2023-09-30T00:29:21.482397+0000","last_peered":"2023-09-30T00:29:21.482397+0000","last_clean":"2023-09-30T00:29:21.482397+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:29:21.482397+0000","last_undegraded":"2023-09-30T00:29:21.482397+0000","last_fullsized":"2023-09-30T00:29:21.482397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:21.482266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:29:21.482266+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:40.264629+0000","scrub_duration":0.001809995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:25.265414+0000","last_change":"2023-09-30T00:29:25.265364+0000","last_active":"2023-09-30T00:29:25.265414+0000","last_peered":"2023-09-30T00:29:25.265414+0000","last_clean":"2023-09-30T00:29:25.265414+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:29:25.265414+0000","last_undegraded":"2023-09-30T00:29:25.265414+0000","last_fullsized":"2023-09-30T00:29:25.265414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:25.265274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:28:02.446802+0000","last_clean_scrub_stamp":"2023-09-30T00:29:25.265274+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:52.997972+0000","scrub_duration":0.0016202670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.288971+0000","last_change":"2023-09-30T00:28:46.895512+0000","last_active":"2023-09-30T00:29:12.288971+0000","last_peered":"2023-09-30T00:29:12.288971+0000","last_clean":"2023-09-30T00:29:12.288971+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:29:12.288971+0000","last_undegraded":"2023-09-30T00:29:12.288971+0000","last_fullsized":"2023-09-30T00:29:12.288971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:46.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:26:28.175176+0000","last_clean_scrub_stamp":"2023-09-30T00:28:46.895402+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:59.201765+0000","scrub_duration":0.0018973250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.049134+0000","last_change":"2023-09-30T00:27:58.265696+0000","last_active":"2023-09-30T00:29:12.049134+0000","last_peered":"2023-09-30T00:29:12.049134+0000","last_clean":"2023-09-30T00:29:12.049134+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:29:12.049134+0000","last_undegraded":"2023-09-30T00:29:12.049134+0000","last_fullsized":"2023-09-30T00:29:12.049134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:27:58.265580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:27:58.265580+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:27.774996+0000","scrub_duration":0.0021068150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":5.4749999999999996,"15min":2.1709999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81100000000000005,"5min":518.15700000000004,"15min":518.15700000000004},"last":0.496},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.95399999999999996,"15min":1.3340000000000001},"min":{"1min":0.318,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":20.98,"15min":20.98},"last":0.67400000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.34000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":36.191000000000003,"5min":7.7039999999999997,"15min":2.944},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":499.12700000000001,"5min":499.12700000000001,"15min":499.12700000000001},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":5.8799999999999999,"15min":2.3740000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82899999999999996,"5min":553.47799999999995,"15min":553.47799999999995},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":17.440000000000001,"15min":18.201000000000001},"min":{"1min":0.432,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78800000000000003,"5min":772.48099999999999,"15min":772.48099999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":47.009,"5min":9.9049999999999994,"15min":3.722},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":742.43200000000002,"5min":742.43200000000002,"15min":742.43200000000002},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":2.431,"15min":1.248},"min":{"1min":0.375,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76500000000000001,"5min":197.946,"15min":197.946},"last":0.42499999999999999},{"interface":"front","average":{"1min":53.822000000000003,"5min":11.23,"15min":4.1870000000000003},"min":{"1min":0.38900000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":852.97199999999998,"5min":852.97199999999998,"15min":852.97199999999998},"last":0.435}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.59499999999999997},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83399999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.374}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":51.947000000000003,"5min":11.907,"15min":4.375},"min":{"1min":0.50800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":821.89999999999998,"5min":821.89999999999998,"15min":821.89999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84499999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.40999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6600,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":94365240,"statfs":{"total":96636764160,"available":96630005760,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":39.131999999999998,"5min":8.3089999999999993,"15min":3.2120000000000002},"min":{"1min":0.498,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":770.97900000000004,"5min":770.97900000000004,"15min":770.97900000000004},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":8.7059999999999995,"15min":16.792000000000002},"min":{"1min":0.47099999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":486.19,"15min":486.19},"last":909.92600000000004},{"interface":"front","average":{"1min":37.652000000000001,"5min":7.9870000000000001,"15min":3.0609999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":742.36199999999997,"5min":742.36199999999997,"15min":742.36199999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":52.683,"5min":24.893000000000001,"15min":8.7520000000000007},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":990.30100000000004,"5min":990.30100000000004,"15min":990.30100000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.63200000000000001},"min":{"1min":0.46600000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88500000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":16.731999999999999,"5min":3.8199999999999998,"15min":1.673},"min":{"1min":0.47799999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":323.27600000000001,"5min":323.27600000000001,"15min":323.27600000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.65100000000000002,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69799999999999995,"5min":11.202,"15min":11.202},"last":0.379}]},{"osd":5,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.39700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70899999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":10.314,"15min":3.8260000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":877.19000000000005,"15min":877.19000000000005},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:28:28 2023","interfaces":[{"interface":"back","average":{"1min":30.800999999999998,"5min":6.6070000000000002,"15min":2.5840000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":606.13499999999999,"5min":606.13499999999999,"15min":606.13499999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":37.212000000000003,"5min":7.9219999999999997,"15min":3.052},"min":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":733.37400000000002,"5min":733.37400000000002,"15min":733.37400000000002},"last":0.438}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":27.849,"5min":12.656000000000001,"15min":4.4989999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":499.25999999999999,"5min":743.42399999999998,"15min":743.42399999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.371,"5min":9.3840000000000003,"15min":3.4180000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61199999999999999,"5min":845.18100000000004,"15min":845.18100000000004},"last":0.56399999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":33.737000000000002,"5min":17.248999999999999,"15min":19.265000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":668.12900000000002,"5min":668.12900000000002,"15min":668.12900000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":10.191000000000001,"15min":19.959},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57499999999999996,"5min":585.68600000000004,"15min":585.68600000000004},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.9780000000000002,"15min":5.5190000000000001},"min":{"1min":0.28000000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.61499999999999999,"5min":148.685,"15min":148.685},"last":0.58099999999999996},{"interface":"front","average":{"1min":27.045000000000002,"5min":5.835,"15min":2.3260000000000001},"min":{"1min":0.318,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":532.24199999999996,"5min":532.24199999999996,"15min":532.24199999999996},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":15.055999999999999,"15min":29.178000000000001},"min":{"1min":0.45900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.84599999999999997,"5min":501.096,"15min":501.096},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":8.9550000000000001,"15min":17.300999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.79400000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":16.785,"15min":18.381},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.82899999999999996,"5min":962.88,"15min":962.88},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":9.0150000000000006,"15min":3.4870000000000001},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":794.64200000000005,"15min":794.64200000000005},"last":0.89200000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":14.795,"15min":28.966999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77000000000000002,"5min":813.57399999999996,"15min":813.57399999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67600000000000005,"15min":0.73399999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.90400000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.44,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71999999999999997,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.66600000000000004,"15min":0.69199999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78900000000000003,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42999999999999999,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.497,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.42599999999999999,"15min":0.41999999999999998},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60599999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.44800000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":41.329000000000001,"5min":8.8130000000000006,"15min":3.2309999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":655.24400000000003,"5min":655.24400000000003,"15min":655.24400000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":12.734,"5min":3.0049999999999999,"15min":1.4830000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":198.53800000000001,"5min":198.53800000000001,"15min":198.53800000000001},"last":0.432},{"interface":"front","average":{"1min":0.434,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.33200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66400000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":5.5990000000000002,"15min":10.590999999999999},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68200000000000005,"5min":275.43200000000002,"15min":275.43200000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.70099999999999996,"15min":0.78200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77600000000000002,"5min":5.1269999999999998,"15min":5.1269999999999998},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":48.792000000000002,"5min":10.356999999999999,"15min":4},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":772.57299999999998,"5min":772.57299999999998,"15min":772.57299999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56100000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.64700000000000002,"15min":0.66000000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.70799999999999996,"5min":1.046,"15min":1.046},"last":0.501}]},{"osd":7,"last update":"Sat Sep 30 00:28:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":14.56,"15min":5.2869999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73199999999999998,"5min":990.16700000000003,"15min":990.16700000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":2.6280000000000001,"15min":1.45},"min":{"1min":0.46600000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.75900000000000001,"5min":198.173,"15min":198.173},"last":0.627}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":29.780000000000001,"5min":18.908000000000001,"15min":24.202999999999999},"min":{"1min":0.34000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":499.13,"5min":586.52599999999995,"15min":586.52599999999995},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.59799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63600000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":9.2240000000000002,"15min":3.4119999999999999},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70199999999999996,"5min":655.45500000000004,"15min":655.45500000000004},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":8.4969999999999999,"15min":3.2869999999999999},"min":{"1min":0.49199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":708.62400000000002,"15min":708.62400000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":7.3339999999999996,"15min":25.207000000000001},"min":{"1min":0.46800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89500000000000002,"5min":536.221,"15min":536.221},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":20.416,"5min":13.345000000000001,"15min":4.9029999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":356.43000000000001,"5min":744.32600000000002,"15min":744.32600000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":9.8510000000000009,"15min":3.75},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90200000000000002,"5min":783.25300000000004,"15min":783.25300000000004},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":83.608999999999995,"5min":17.245999999999999,"15min":6.2199999999999998},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":678.30799999999999,"5min":678.30799999999999,"15min":678.30799999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.68799999999999994},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.60399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.64000000000000001,"15min":0.84799999999999998},"min":{"1min":0.28899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.433},{"interface":"front","average":{"1min":0.39800000000000002,"5min":18.452999999999999,"15min":35.728000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":733.85000000000002,"15min":733.85000000000002},"last":0.42199999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":15.871,"5min":15.871,"15min":15.871},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.84199999999999997,"15min":1.264},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53700000000000003,"5min":9.157,"15min":9.157},"last":986.77999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.68100000000000005,"15min":0.93700000000000006},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60399999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.54700000000000004,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57799999999999996,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.40100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":4.1269999999999998,"15min":2.1869999999999998},"min":{"1min":0.46500000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.83899999999999997,"5min":277.21499999999997,"15min":277.21499999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":16.873999999999999,"5min":3.9790000000000001,"15min":1.8899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":309.69099999999997,"5min":309.69099999999997,"15min":309.69099999999997},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":38.488999999999997,"5min":8.4239999999999995,"15min":3.581},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":668.34500000000003,"5min":668.34500000000003,"15min":668.34500000000003},"last":444.73000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.89000000000000001,"15min":1.1299999999999999},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86499999999999999,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.80000000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.92500000000000004,"5min":3.5289999999999999,"15min":3.5289999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.86699999999999999,"15min":1.091},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.879,"5min":9.0950000000000006,"15min":9.0950000000000006},"last":0.60199999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:28:23 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.77400000000000002,"15min":0.88400000000000001},"min":{"1min":0.433,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90100000000000002,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.85499999999999998,"15min":0.871},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79800000000000004,"5min":11.789999999999999,"15min":11.789999999999999},"last":0.65800000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":7.1230000000000002,"15min":13.659000000000001},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":360.43599999999998,"15min":360.43599999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.83199999999999996,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":11.308999999999999,"15min":4.1859999999999999},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81100000000000005,"5min":909.94100000000003,"15min":909.94100000000003},"last":0.77300000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63400000000000001,"15min":0.64500000000000002},"min":{"1min":0.48299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80100000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.749},{"interface":"front","average":{"1min":0.58799999999999997,"5min":12.667,"15min":4.6589999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":963.85500000000002,"15min":963.85500000000002},"last":0.66200000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.8029999999999999,"15min":6.9450000000000003},"min":{"1min":0.53400000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94599999999999995,"5min":182.018,"15min":182.018},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":7.734,"15min":3.024},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90900000000000003,"5min":565.77200000000005,"15min":565.77200000000005},"last":0.71299999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.34799999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":21.582999999999998,"5min":4.7290000000000001,"15min":1.917},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":463.30700000000002,"5min":463.30700000000002,"15min":463.30700000000002},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":6.194,"15min":2.4159999999999999},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83999999999999997,"5min":453.32400000000001,"15min":453.32400000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":13.493,"15min":4.8789999999999996},"min":{"1min":0.40799999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89000000000000001,"5min":596.84500000000003,"15min":596.84500000000003},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Sep 30 00:28:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":10.733000000000001,"15min":20.893999999999998},"min":{"1min":0.40100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92300000000000004,"5min":585.08000000000004,"15min":585.08000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":36.701999999999998,"5min":21.587,"15min":30.317},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.38},"max":{"1min":794.49000000000001,"5min":794.49000000000001,"15min":794.49000000000001},"last":0.80900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":17.228999999999999,"15min":6.1539999999999999},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76600000000000001,"5min":954.57299999999998,"15min":954.57299999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":5.3689999999999998,"15min":2.1989999999999998},"min":{"1min":0.437,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82899999999999996,"5min":453.76100000000002,"15min":453.76100000000002},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":3.3879999999999999,"5min":23.856999999999999,"15min":40.771000000000001},"min":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":51.552999999999997,"5min":973.71299999999997,"15min":973.71299999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":8.8759999999999994,"15min":3.3639999999999999},"min":{"1min":0.38700000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71599999999999997,"5min":788.19899999999996,"15min":788.19899999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70099999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":12.221,"5min":2.9159999999999999,"15min":1.383},"min":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":198.43000000000001,"5min":198.43000000000001,"15min":198.43000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.628,"15min":0.65100000000000002},"min":{"1min":0.51200000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69299999999999995,"5min":1.292,"15min":1.292},"last":0.623}]},{"osd":4,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48099999999999998,"15min":0.497},"min":{"1min":0.35499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.52500000000000002},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72799999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":15.48,"15min":30.431999999999999},"min":{"1min":0.441,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76400000000000001,"5min":784.48299999999995,"15min":784.48299999999995},"last":0.42599999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:28:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.55600000000000005},"min":{"1min":0.35999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70499999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":43.857999999999997,"5min":25.468,"15min":35.968000000000004},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":736.56700000000001,"5min":853.28200000000004,"15min":853.28200000000004},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:29:29.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:29:29.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:29.931 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"fsid":"38d43cc8-1041-4ca0-9db0-4fe8f50adae4","created":"2023-09-30T00:24:08.097432+0000","modified":"2023-09-30T00:29:18.053662+0000","last_up_change":"2023-09-30T00:24:16.843463+0000","last_in_change":"2023-09-30T00:24:10.540428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:24:17.820431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:24:19.342552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"856896a0-c275-4afb-9fc8-0ee850f2ef48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":88112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":88112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":88112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":88112}]},"public_addr":"172.21.15.31:6804/88112","cluster_addr":"172.21.15.31:6805/88112","heartbeat_back_addr":"172.21.15.31:6807/88112","heartbeat_front_addr":"172.21.15.31:6806/88112","state":["exists","up"]},{"osd":1,"uuid":"e4e24806-a2e7-47e0-b0bf-8efce59fd6ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":88103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6801","nonce":88103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":88103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":88103}]},"public_addr":"172.21.15.31:6800/88103","cluster_addr":"172.21.15.31:6801/88103","heartbeat_back_addr":"172.21.15.31:6803/88103","heartbeat_front_addr":"172.21.15.31:6802/88103","state":["exists","up"]},{"osd":2,"uuid":"c05d81e7-d71d-4b9d-8990-2c33c75bc836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":88113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":88113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":88113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":88113}]},"public_addr":"172.21.15.31:6808/88113","cluster_addr":"172.21.15.31:6809/88113","heartbeat_back_addr":"172.21.15.31:6811/88113","heartbeat_front_addr":"172.21.15.31:6810/88113","state":["exists","up"]},{"osd":3,"uuid":"bb2e1b0d-ce9b-489d-b5c7-9ffdcc615e49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":88114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":88114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6815","nonce":88114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":88114}]},"public_addr":"172.21.15.31:6812/88114","cluster_addr":"172.21.15.31:6813/88114","heartbeat_back_addr":"172.21.15.31:6815/88114","heartbeat_front_addr":"172.21.15.31:6814/88114","state":["exists","up"]},{"osd":4,"uuid":"61ec23b4-115e-41a9-a06b-be2c06e89ba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":88052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6805","nonce":88052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6807","nonce":88052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":88052}]},"public_addr":"172.21.15.146:6804/88052","cluster_addr":"172.21.15.146:6805/88052","heartbeat_back_addr":"172.21.15.146:6807/88052","heartbeat_front_addr":"172.21.15.146:6806/88052","state":["exists","up"]},{"osd":5,"uuid":"3ed6d497-61de-4612-bc24-8eb8052076a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":88051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6801","nonce":88051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6803","nonce":88051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":88051}]},"public_addr":"172.21.15.146:6800/88051","cluster_addr":"172.21.15.146:6801/88051","heartbeat_back_addr":"172.21.15.146:6803/88051","heartbeat_front_addr":"172.21.15.146:6802/88051","state":["exists","up"]},{"osd":6,"uuid":"b3c1cca2-1530-4e60-a4f0-3bedd6f433eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":88081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":88081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6811","nonce":88081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":88081}]},"public_addr":"172.21.15.146:6808/88081","cluster_addr":"172.21.15.146:6809/88081","heartbeat_back_addr":"172.21.15.146:6811/88081","heartbeat_front_addr":"172.21.15.146:6810/88081","state":["exists","up"]},{"osd":7,"uuid":"46762b2d-2b3b-4a90-8577-3b9278c07650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":88082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6813","nonce":88082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":88082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":88082}]},"public_addr":"172.21.15.146:6812/88082","cluster_addr":"172.21.15.146:6813/88082","heartbeat_back_addr":"172.21.15.146:6815/88082","heartbeat_front_addr":"172.21.15.146:6814/88082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:29:00.951191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:59.454171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:42.329368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:58.015825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:29:17.764025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:13.464969+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:28:14.332961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:29:11.581585+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:29:30.946 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T00:29:30.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:31.112 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:31.112 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:31.112 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:31.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T00:29:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T00:29:31.464 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T00:29:31.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:31.616 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:31.617 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:31.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T00:29:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T00:29:31.972 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T00:29:31.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:32.124 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:32.124 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:32.124 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:32.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T00:29:32.465 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T00:29:32.476 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T00:29:32.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:32.630 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:32.631 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:32.631 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:32.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T00:29:32.972 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T00:29:32.983 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T00:29:32.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:33.137 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:33.137 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:33.138 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:33.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T00:29:33.480 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T00:29:33.490 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T00:29:33.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:33.644 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:33.644 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:33.644 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:33.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T00:29:33.984 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T00:29:33.995 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T00:29:33.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:34.148 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:34.148 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:34.148 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:34.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T00:29:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T00:29:34.500 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T00:29:34.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:29:34.654 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-09-30T00:29:34.654 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:29:34.654 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-09-30T00:29:34.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T00:29:34.995 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T00:29:35.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:29:35.333 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:35.335 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:29:35.346 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2023-09-30T00:29:33.828269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":48152,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":43520,"kb_avail":754926568,"statfs":{"total":773094113280,"available":773044805632,"internally_reserved":0,"allocated":4677632,"data_stored":2956344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44564480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002474"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.048852+0000","last_change":"2023-09-30T00:28:18.718832+0000","last_active":"2023-09-30T00:29:12.048852+0000","last_peered":"2023-09-30T00:29:12.048852+0000","last_clean":"2023-09-30T00:29:12.048852+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:29:12.048852+0000","last_undegraded":"2023-09-30T00:29:12.048852+0000","last_fullsized":"2023-09-30T00:29:12.048852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:18.718735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:18.718735+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:28.151748+0000","scrub_duration":0.001786316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:30.917078+0000","last_change":"2023-09-30T00:29:30.917023+0000","last_active":"2023-09-30T00:29:30.917078+0000","last_peered":"2023-09-30T00:29:30.917078+0000","last_clean":"2023-09-30T00:29:30.917078+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:29:30.917078+0000","last_undegraded":"2023-09-30T00:29:30.917078+0000","last_fullsized":"2023-09-30T00:29:30.917078+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:30.916918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:29:30.916918+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0020173779999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944502+0000","last_change":"2023-09-30T00:28:21.511933+0000","last_active":"2023-09-30T00:29:12.944502+0000","last_peered":"2023-09-30T00:29:12.944502+0000","last_clean":"2023-09-30T00:29:12.944502+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:29:12.944502+0000","last_undegraded":"2023-09-30T00:29:12.944502+0000","last_fullsized":"2023-09-30T00:29:12.944502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:21.511819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:21.511819+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:41.914937+0000","scrub_duration":0.0016831120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.944516+0000","last_change":"2023-09-30T00:28:17.446760+0000","last_active":"2023-09-30T00:29:12.944516+0000","last_peered":"2023-09-30T00:29:12.944516+0000","last_clean":"2023-09-30T00:29:12.944516+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:29:12.944516+0000","last_undegraded":"2023-09-30T00:29:12.944516+0000","last_fullsized":"2023-09-30T00:29:12.944516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:17.446653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:28:17.446653+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:39.685841+0000","scrub_duration":0.0021246559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":232,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:20.841814+0000","last_change":"2023-09-30T00:29:20.841771+0000","last_active":"2023-09-30T00:29:20.841814+0000","last_peered":"2023-09-30T00:29:20.841814+0000","last_clean":"2023-09-30T00:29:20.841814+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:29:20.841814+0000","last_undegraded":"2023-09-30T00:29:20.841814+0000","last_fullsized":"2023-09-30T00:29:20.841814+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2023-09-30T00:29:20.841644+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2023-09-30T00:25:30.067930+0000","last_clean_scrub_stamp":"2023-09-30T00:29:20.841644+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:48.971707+0000","scrub_duration":0.0023337359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:21.482397+0000","last_change":"2023-09-30T00:29:21.482365+0000","last_active":"2023-09-30T00:29:21.482397+0000","last_peered":"2023-09-30T00:29:21.482397+0000","last_clean":"2023-09-30T00:29:21.482397+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:29:21.482397+0000","last_undegraded":"2023-09-30T00:29:21.482397+0000","last_fullsized":"2023-09-30T00:29:21.482397+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:21.482266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:24:19.872849+0000","last_clean_scrub_stamp":"2023-09-30T00:29:21.482266+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:40.264629+0000","scrub_duration":0.001809995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:25.265414+0000","last_change":"2023-09-30T00:29:25.265364+0000","last_active":"2023-09-30T00:29:25.265414+0000","last_peered":"2023-09-30T00:29:25.265414+0000","last_clean":"2023-09-30T00:29:25.265414+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:29:25.265414+0000","last_undegraded":"2023-09-30T00:29:25.265414+0000","last_fullsized":"2023-09-30T00:29:25.265414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:25.265274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:28:02.446802+0000","last_clean_scrub_stamp":"2023-09-30T00:29:25.265274+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:52.997972+0000","scrub_duration":0.0016202670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-09-30T00:29:12.288971+0000","last_change":"2023-09-30T00:28:46.895512+0000","last_active":"2023-09-30T00:29:12.288971+0000","last_peered":"2023-09-30T00:29:12.288971+0000","last_clean":"2023-09-30T00:29:12.288971+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:29:12.288971+0000","last_undegraded":"2023-09-30T00:29:12.288971+0000","last_fullsized":"2023-09-30T00:29:12.288971+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:28:46.895402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:26:28.175176+0000","last_clean_scrub_stamp":"2023-09-30T00:28:46.895402+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:29:59.201765+0000","scrub_duration":0.0018973250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:29:28.993865+0000","last_change":"2023-09-30T00:29:28.993804+0000","last_active":"2023-09-30T00:29:28.993865+0000","last_peered":"2023-09-30T00:29:28.993865+0000","last_clean":"2023-09-30T00:29:28.993865+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:29:28.993865+0000","last_undegraded":"2023-09-30T00:29:28.993865+0000","last_fullsized":"2023-09-30T00:29:28.993865+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:28.993678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:28.993678+0000","last_clean_scrub_stamp":"2023-09-30T00:29:28.993678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.001632957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":51.268999999999998,"5min":15.625,"15min":5.5549999999999997},"min":{"1min":0.318,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":862.96199999999999,"5min":862.96199999999999,"15min":862.96199999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.75900000000000001,"15min":1.2689999999999999},"min":{"1min":0.39800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83599999999999997,"5min":20.98,"15min":20.98},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":58.442999999999998,"5min":12.125,"15min":4.3949999999999996},"min":{"1min":0.46600000000000003,"5min":0.318,"15min":0.318},"max":{"1min":984.74699999999996,"5min":984.74699999999996,"15min":984.74699999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":7.7030000000000003,"15min":2.9430000000000001},"min":{"1min":0.38200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80900000000000005,"5min":499.12700000000001,"15min":499.12700000000001},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.47199999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":5.8689999999999998,"15min":2.371},"min":{"1min":0.44,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77800000000000002,"5min":553.47799999999995,"15min":553.47799999999995},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":32.783000000000001,"5min":20.280000000000001,"15min":19.148},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":548,"5min":772.48099999999999,"15min":772.48099999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.622,"5min":9.9030000000000005,"15min":3.7210000000000001},"min":{"1min":0.503,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97299999999999998,"5min":742.43200000000002,"15min":742.43200000000002},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":2.3919999999999999,"15min":1.2350000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59299999999999997,"5min":197.946,"15min":197.946},"last":0.42999999999999999},{"interface":"front","average":{"1min":30.745000000000001,"5min":17.245999999999999,"15min":6.1920000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":514.80399999999997,"5min":852.97199999999998,"15min":852.97199999999998},"last":0.44700000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70299999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.47299999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":36.201000000000001,"5min":19.024999999999999,"15min":6.7480000000000002},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":606.86599999999999,"5min":821.89999999999998,"15min":821.89999999999998},"last":0.377},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70199999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.504}]}]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.77600000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":8.2970000000000006,"15min":3.2080000000000002},"min":{"1min":0.495,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":770.97900000000004,"15min":770.97900000000004},"last":0.57899999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":51.109999999999999,"5min":14.760999999999999,"15min":18.811},"min":{"1min":0.39500000000000002,"5min":0.316,"15min":0.316},"max":{"1min":909.92600000000004,"5min":909.92600000000004,"15min":909.92600000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":27.942,"5min":13.455,"15min":4.8840000000000003},"min":{"1min":0.42199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":492.471,"5min":742.36199999999997,"15min":742.36199999999997},"last":492.471}]},{"osd":2,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":24.881,"15min":8.7479999999999993},"min":{"1min":0.49399999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":990.30100000000004,"15min":990.30100000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":44.832999999999998,"5min":9.4459999999999997,"15min":3.5779999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":840.41200000000003,"5min":840.41200000000003,"15min":840.41200000000003},"last":0.53700000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.8239999999999998,"15min":1.6739999999999999},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.755,"5min":323.27600000000001,"15min":323.27600000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.621},"min":{"1min":0.52600000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.82299999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52600000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.502},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.64300000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":39.576000000000001,"5min":8.4540000000000006,"15min":3.1909999999999998},"min":{"1min":0.36199999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":742.05999999999995,"5min":742.05999999999995,"15min":742.05999999999995},"last":0.71599999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":25.481999999999999,"5min":5.5170000000000003,"15min":2.2090000000000001},"min":{"1min":0.39900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":474.51900000000001,"5min":474.51900000000001,"15min":474.51900000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":37.012999999999998,"5min":17.600000000000001,"15min":6.2549999999999999},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":693.37699999999995,"5min":877.19000000000005,"15min":877.19000000000005},"last":0.69099999999999995}]},{"osd":7,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":34.536999999999999,"5min":13.4,"15min":4.8479999999999999},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":612.77599999999995,"5min":612.77599999999995,"15min":612.77599999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":7.9139999999999997,"15min":3.0489999999999999},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74099999999999999,"5min":733.37400000000002,"15min":733.37400000000002},"last":0.66900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":3.8180000000000001,"5min":13.335000000000001,"15min":4.7249999999999996},"min":{"1min":0.33000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":64.507000000000005,"5min":743.42399999999998,"15min":743.42399999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":9.3870000000000005,"15min":3.419},"min":{"1min":0.27800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.57799999999999996,"5min":845.18100000000004,"15min":845.18100000000004},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":13.286,"15min":17.943999999999999},"min":{"1min":0.33400000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.58899999999999997,"5min":668.12900000000002,"15min":668.12900000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":5.319,"15min":18.335000000000001},"min":{"1min":0.309,"5min":0.248,"15min":0.248},"max":{"1min":0.61399999999999999,"5min":585.68600000000004,"15min":585.68600000000004},"last":0.32300000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.7150000000000001,"15min":5.0979999999999999},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65200000000000002,"5min":148.685,"15min":148.685},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":5.8300000000000001,"15min":2.3239999999999998},"min":{"1min":0.37,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64500000000000002,"5min":532.24199999999996,"15min":532.24199999999996},"last":0.36199999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":7.9459999999999997,"15min":26.808},"min":{"1min":0.46100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86899999999999999,"5min":501.096,"15min":501.096},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":4.7969999999999997,"15min":15.914999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82099999999999995,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.62}]},{"osd":5,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":35.369999999999997,"5min":20.023,"15min":19.460999999999999},"min":{"1min":0.52600000000000002,"5min":0.435,"15min":0.435},"max":{"1min":694.34900000000005,"5min":962.88,"15min":962.88},"last":0.66200000000000003},{"interface":"front","average":{"1min":35.598999999999997,"5min":15.99,"15min":5.8120000000000003},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":699.37599999999998,"5min":794.64200000000005,"15min":794.64200000000005},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":7.7240000000000002,"15min":26.609999999999999},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84999999999999998,"5min":813.57399999999996,"15min":813.57399999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.66600000000000004,"15min":0.73099999999999998},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65100000000000002,"15min":0.69499999999999995},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81299999999999994,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.67100000000000004,"15min":0.69399999999999995},"min":{"1min":0.51000000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88200000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.58899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6664,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365176,"statfs":{"total":96636764160,"available":96629940224,"internally_reserved":0,"allocated":917504,"data_stored":700123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.30299999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.64900000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63500000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.61299999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":8.8149999999999995,"15min":3.2309999999999999},"min":{"1min":0.27800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.67800000000000005,"5min":655.24400000000003,"15min":655.24400000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":2.9449999999999998,"15min":1.4630000000000001},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60299999999999998,"5min":198.53800000000001,"15min":198.53800000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.29899999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65200000000000002,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":3.1080000000000001,"15min":9.7599999999999998},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92300000000000004,"5min":275.43200000000002,"15min":275.43200000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":16.745999999999999,"5min":3.8849999999999998,"15min":1.843},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":322.64999999999998,"5min":322.64999999999998,"15min":322.64999999999998},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":10.323,"15min":3.9889999999999999},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92500000000000004,"5min":772.57299999999998,"15min":772.57299999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64300000000000002,"15min":0.66200000000000003},"min":{"1min":0.47199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90800000000000003,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65000000000000002,"15min":0.64600000000000002},"min":{"1min":0.442,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97099999999999997,"5min":1.046,"15min":1.046},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":39.786999999999999,"5min":22.387,"15min":7.8959999999999999},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":822.63300000000004,"5min":990.16700000000003,"15min":990.16700000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":2.5939999999999999,"15min":1.4390000000000001},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93799999999999994,"5min":198.173,"15min":198.173},"last":0.55800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.311},{"interface":"front","average":{"1min":29.780000000000001,"5min":18.908000000000001,"15min":24.202999999999999},"min":{"1min":0.34000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":499.13,"5min":586.52599999999995,"15min":586.52599999999995},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.59799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63600000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.501,"5min":9.2240000000000002,"15min":3.4119999999999999},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70199999999999996,"5min":655.45500000000004,"15min":655.45500000000004},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":8.4969999999999999,"15min":3.2869999999999999},"min":{"1min":0.49199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":708.62400000000002,"15min":708.62400000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":7.3339999999999996,"15min":25.207000000000001},"min":{"1min":0.46800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89500000000000002,"5min":536.221,"15min":536.221},"last":0.58699999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":20.416,"5min":13.345000000000001,"15min":4.9029999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":356.43000000000001,"5min":744.32600000000002,"15min":744.32600000000002},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.503},{"interface":"front","average":{"1min":0.63900000000000001,"5min":9.8510000000000009,"15min":3.75},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90200000000000002,"5min":783.25300000000004,"15min":783.25300000000004},"last":0.73699999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":83.608999999999995,"5min":17.245999999999999,"15min":6.2199999999999998},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":678.30799999999999,"5min":678.30799999999999,"15min":678.30799999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.68799999999999994},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.52800000000000002,"15min":0.81100000000000005},"min":{"1min":0.29499999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52900000000000003,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":9.6630000000000003,"15min":32.798000000000002},"min":{"1min":0.316,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.58699999999999997,"5min":733.85000000000002,"15min":733.85000000000002},"last":0.46100000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.30099999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63200000000000001,"5min":15.871,"15min":15.871},"last":0.39600000000000002},{"interface":"front","average":{"1min":88.762,"5min":18.300000000000001,"15min":7.0839999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":986.77999999999997,"5min":986.77999999999997,"15min":986.77999999999997},"last":0.373}]},{"osd":2,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.55700000000000005,"15min":0.89600000000000002},"min":{"1min":0.33200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60799999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.504,"15min":0.622},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67100000000000004,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.442}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":4.0019999999999998,"15min":2.145},"min":{"1min":0.46200000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.77200000000000002,"5min":277.21499999999997,"15min":277.21499999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":3.931,"15min":1.8740000000000001},"min":{"1min":0.47799999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83799999999999997,"5min":309.69099999999997,"15min":309.69099999999997},"last":0.51000000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":29.396000000000001,"5min":14.071,"15min":5.4630000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":444.73000000000002,"5min":668.34500000000003,"15min":668.34500000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.76300000000000001,"15min":1.087},"min":{"1min":0.48099999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.88100000000000001,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.55900000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.68200000000000005,"15min":0.78400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84899999999999998,"5min":3.5289999999999999,"15min":3.5289999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.749,"15min":1.052},"min":{"1min":0.51200000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.76600000000000001,"5min":9.0950000000000006,"15min":9.0950000000000006},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.71299999999999997,"15min":0.86299999999999999},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90500000000000003,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":4.4299999999999997,"5min":1.5649999999999999,"15min":1.107},"min":{"1min":0.51100000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":68.840999999999994,"5min":68.840999999999994,"15min":68.840999999999994},"last":0.56599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.39100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":3.8479999999999999,"15min":12.568},"min":{"1min":0.36799999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69599999999999995,"5min":360.43599999999998,"15min":360.43599999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71799999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":578.43499999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":11.301,"15min":4.1829999999999998},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83199999999999996,"5min":909.94100000000003,"15min":909.94100000000003},"last":0.57299999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.39400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.749,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.62,"5min":12.66,"15min":4.657},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76000000000000001,"5min":963.85500000000002,"15min":963.85500000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":71.781000000000006,"5min":16.456,"15min":11.163},"min":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":680.40800000000002,"5min":680.40800000000002,"15min":680.40800000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":7.7240000000000002,"15min":3.02},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75800000000000001,"5min":565.77200000000005,"15min":565.77200000000005},"last":0.45600000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.503,"15min":0.501},"min":{"1min":0.373,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67300000000000004,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.496},{"interface":"front","average":{"1min":0.52800000000000002,"5min":4.7329999999999997,"15min":1.9179999999999999},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69799999999999995,"5min":463.30700000000002,"15min":463.30700000000002},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":6.1950000000000003,"15min":2.4169999999999998},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79200000000000004,"5min":453.32400000000001,"15min":453.32400000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":13.493,"15min":4.8789999999999996},"min":{"1min":0.45400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73199999999999998,"5min":596.84500000000003,"15min":596.84500000000003},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":5.6520000000000001,"15min":19.199999999999999},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80100000000000005,"5min":585.08000000000004,"15min":585.08000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":14.768000000000001,"15min":28.044},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.80900000000000005,"5min":794.49000000000001,"15min":794.49000000000001},"last":0.51500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5632,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366208,"statfs":{"total":96636764160,"available":96630996992,"internally_reserved":0,"allocated":385024,"data_stored":171195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":17.231000000000002,"15min":6.1550000000000002},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.117,"5min":954.57299999999998,"15min":954.57299999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":5.3689999999999998,"15min":2.1989999999999998},"min":{"1min":0.46600000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":453.76100000000002,"15min":453.76100000000002},"last":0.38100000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":14.135999999999999,"15min":37.530999999999999},"min":{"1min":0.41099999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77100000000000002,"5min":973.71299999999997,"15min":973.71299999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":8.8819999999999997,"15min":3.3660000000000001},"min":{"1min":0.42799999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.105,"5min":788.19899999999996,"15min":788.19899999999996},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":2.915,"15min":1.383},"min":{"1min":0.42999999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.079,"5min":198.43000000000001,"15min":198.43000000000001},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.502,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77000000000000002,"5min":1.292,"15min":1.292},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":43.25,"5min":9.0299999999999994,"15min":3.3460000000000001},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.316},"max":{"1min":642.40800000000002,"5min":642.40800000000002,"15min":642.40800000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.495,"15min":0.52100000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.624,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.495}]},{"osd":6,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69199999999999995,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":8.0060000000000002,"15min":27.940000000000001},"min":{"1min":0.41899999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67800000000000005,"5min":784.48299999999995,"15min":784.48299999999995},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52200000000000002,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":17.327000000000002,"15min":33.253999999999998},"min":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70399999999999996,"5min":853.28200000000004,"15min":853.28200000000004},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:29:35.354 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T00:28:18.718735+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=28, tm_sec=18, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.354 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T00:28:21.511819+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=28, tm_sec=21, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.354 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T00:28:17.446653+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=28, tm_sec=17, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.354 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T00:29:20.841644+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=20, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.354 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T00:29:21.482266+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=21, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.354 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T00:29:25.265274+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=25, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.355 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T00:28:46.895402+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=28, tm_sec=46, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.355 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T00:29:28.993678+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=28, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=29, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:29:35.355 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T00:29:55.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:29:55.694 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-30T00:29:55.696 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-30T00:29:55.707 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2023-09-30T00:29:53.832483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":48408,"kb_used_data":4760,"kb_used_omap":0,"kb_used_meta":43584,"kb_avail":754926312,"statfs":{"total":773094113280,"available":773044543488,"internally_reserved":0,"allocated":4874240,"data_stored":3060216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44630016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002138"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:39.131644+0000","last_change":"2023-09-30T00:29:34.593711+0000","last_active":"2023-09-30T00:29:39.131644+0000","last_peered":"2023-09-30T00:29:39.131644+0000","last_clean":"2023-09-30T00:29:39.131644+0000","last_became_active":"2023-09-30T00:24:20.901155+0000","last_became_peered":"2023-09-30T00:24:20.901155+0000","last_unstale":"2023-09-30T00:29:39.131644+0000","last_undegraded":"2023-09-30T00:29:39.131644+0000","last_fullsized":"2023-09-30T00:29:39.131644+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:34.593572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:34.593572+0000","last_clean_scrub_stamp":"2023-09-30T00:29:34.593572+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:58.816548+0000","scrub_duration":0.0019446439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":92,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:39.126876+0000","last_change":"2023-09-30T00:29:38.994685+0000","last_active":"2023-09-30T00:29:39.126876+0000","last_peered":"2023-09-30T00:29:39.126876+0000","last_clean":"2023-09-30T00:29:39.126876+0000","last_became_active":"2023-09-30T00:24:21.129413+0000","last_became_peered":"2023-09-30T00:24:21.129413+0000","last_unstale":"2023-09-30T00:29:39.126876+0000","last_undegraded":"2023-09-30T00:29:39.126876+0000","last_fullsized":"2023-09-30T00:29:39.126876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:38.994579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:38.994579+0000","last_clean_scrub_stamp":"2023-09-30T00:29:38.994579+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:31:08.764475+0000","scrub_duration":0.0020319079999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":82,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:39.616533+0000","last_change":"2023-09-30T00:29:39.616469+0000","last_active":"2023-09-30T00:29:39.616533+0000","last_peered":"2023-09-30T00:29:39.616533+0000","last_clean":"2023-09-30T00:29:39.616533+0000","last_became_active":"2023-09-30T00:24:21.130577+0000","last_became_peered":"2023-09-30T00:24:21.130577+0000","last_unstale":"2023-09-30T00:29:39.616533+0000","last_undegraded":"2023-09-30T00:29:39.616533+0000","last_fullsized":"2023-09-30T00:29:39.616533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:39.616376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:39.616376+0000","last_clean_scrub_stamp":"2023-09-30T00:29:39.616376+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:44.461351+0000","scrub_duration":0.0021977910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":82,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:39.616082+0000","last_change":"2023-09-30T00:29:36.615866+0000","last_active":"2023-09-30T00:29:39.616082+0000","last_peered":"2023-09-30T00:29:39.616082+0000","last_clean":"2023-09-30T00:29:39.616082+0000","last_became_active":"2023-09-30T00:24:21.129470+0000","last_became_peered":"2023-09-30T00:24:21.129470+0000","last_unstale":"2023-09-30T00:29:39.616082+0000","last_undegraded":"2023-09-30T00:29:39.616082+0000","last_fullsized":"2023-09-30T00:29:39.616082+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:36.615769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:36.615769+0000","last_clean_scrub_stamp":"2023-09-30T00:29:36.615769+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:31:05.062822+0000","scrub_duration":0.0023596250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":246,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:39.615961+0000","last_change":"2023-09-30T00:29:35.583637+0000","last_active":"2023-09-30T00:29:39.615961+0000","last_peered":"2023-09-30T00:29:39.615961+0000","last_clean":"2023-09-30T00:29:39.615961+0000","last_became_active":"2023-09-30T00:24:19.331791+0000","last_became_peered":"2023-09-30T00:24:19.331791+0000","last_unstale":"2023-09-30T00:29:39.615961+0000","last_undegraded":"2023-09-30T00:29:39.615961+0000","last_fullsized":"2023-09-30T00:29:39.615961+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2023-09-30T00:29:35.583524+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2023-09-30T00:29:35.583524+0000","last_clean_scrub_stamp":"2023-09-30T00:29:35.583524+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:50.030659+0000","scrub_duration":0.013357875999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":91,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:40.523516+0000","last_change":"2023-09-30T00:29:37.547992+0000","last_active":"2023-09-30T00:29:40.523516+0000","last_peered":"2023-09-30T00:29:40.523516+0000","last_clean":"2023-09-30T00:29:40.523516+0000","last_became_active":"2023-09-30T00:24:20.900161+0000","last_became_peered":"2023-09-30T00:24:20.900161+0000","last_unstale":"2023-09-30T00:29:40.523516+0000","last_undegraded":"2023-09-30T00:29:40.523516+0000","last_fullsized":"2023-09-30T00:29:40.523516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:37.547899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:37.547899+0000","last_clean_scrub_stamp":"2023-09-30T00:29:37.547899+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:46.695028+0000","scrub_duration":0.0019387059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:40.523452+0000","last_change":"2023-09-30T00:29:33.320439+0000","last_active":"2023-09-30T00:29:40.523452+0000","last_peered":"2023-09-30T00:29:40.523452+0000","last_clean":"2023-09-30T00:29:40.523452+0000","last_became_active":"2023-09-30T00:24:21.130633+0000","last_became_peered":"2023-09-30T00:24:21.130633+0000","last_unstale":"2023-09-30T00:29:40.523452+0000","last_undegraded":"2023-09-30T00:29:40.523452+0000","last_fullsized":"2023-09-30T00:29:40.523452+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:33.320354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:33.320354+0000","last_clean_scrub_stamp":"2023-09-30T00:29:33.320354+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:45.916934+0000","scrub_duration":0.001692755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":91,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:40.523519+0000","last_change":"2023-09-30T00:29:35.515171+0000","last_active":"2023-09-30T00:29:40.523519+0000","last_peered":"2023-09-30T00:29:40.523519+0000","last_clean":"2023-09-30T00:29:40.523519+0000","last_became_active":"2023-09-30T00:24:20.900431+0000","last_became_peered":"2023-09-30T00:24:20.900431+0000","last_unstale":"2023-09-30T00:29:40.523519+0000","last_undegraded":"2023-09-30T00:29:40.523519+0000","last_fullsized":"2023-09-30T00:29:40.523519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:35.515041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:35.515041+0000","last_clean_scrub_stamp":"2023-09-30T00:29:35.515041+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:30:58.841957+0000","scrub_duration":0.001940532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:29:39.947086+0000","last_change":"2023-09-30T00:29:39.947034+0000","last_active":"2023-09-30T00:29:39.947086+0000","last_peered":"2023-09-30T00:29:39.947086+0000","last_clean":"2023-09-30T00:29:39.947086+0000","last_became_active":"2023-09-30T00:24:21.130607+0000","last_became_peered":"2023-09-30T00:24:21.130607+0000","last_unstale":"2023-09-30T00:29:39.947086+0000","last_undegraded":"2023-09-30T00:29:39.947086+0000","last_fullsized":"2023-09-30T00:29:39.947086+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:29:39.946934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:29:39.946934+0000","last_clean_scrub_stamp":"2023-09-30T00:29:39.946934+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:31:05.761205+0000","scrub_duration":0.0019170509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6688,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365152,"statfs":{"total":96636764160,"available":96629915648,"internally_reserved":0,"allocated":942080,"data_stored":713107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":51.268999999999998,"5min":15.625,"15min":5.5549999999999997},"min":{"1min":0.318,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":862.96199999999999,"5min":862.96199999999999,"15min":862.96199999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.75900000000000001,"15min":1.2689999999999999},"min":{"1min":0.39800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83599999999999997,"5min":20.98,"15min":20.98},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":58.442999999999998,"5min":12.125,"15min":4.3949999999999996},"min":{"1min":0.46600000000000003,"5min":0.318,"15min":0.318},"max":{"1min":984.74699999999996,"5min":984.74699999999996,"15min":984.74699999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":7.7030000000000003,"15min":2.9430000000000001},"min":{"1min":0.38200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80900000000000005,"5min":499.12700000000001,"15min":499.12700000000001},"last":0.503}]},{"osd":2,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.47199999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":5.8689999999999998,"15min":2.371},"min":{"1min":0.44,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77800000000000002,"5min":553.47799999999995,"15min":553.47799999999995},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":32.783000000000001,"5min":20.280000000000001,"15min":19.148},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":548,"5min":772.48099999999999,"15min":772.48099999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.622,"5min":9.9030000000000005,"15min":3.7210000000000001},"min":{"1min":0.503,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97299999999999998,"5min":742.43200000000002,"15min":742.43200000000002},"last":0.65800000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":2.3919999999999999,"15min":1.2350000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59299999999999997,"5min":197.946,"15min":197.946},"last":0.59499999999999997},{"interface":"front","average":{"1min":30.745000000000001,"5min":17.245999999999999,"15min":6.1920000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":514.80399999999997,"5min":852.97199999999998,"15min":852.97199999999998},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70299999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57199999999999995,"15min":0.59199999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:29:25 2023","interfaces":[{"interface":"back","average":{"1min":36.201000000000001,"5min":19.024999999999999,"15min":6.7480000000000002},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":606.86599999999999,"5min":821.89999999999998,"15min":821.89999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70199999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.63200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6688,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365152,"statfs":{"total":96636764160,"available":96629915648,"internally_reserved":0,"allocated":942080,"data_stored":713107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.77600000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":8.2970000000000006,"15min":3.2080000000000002},"min":{"1min":0.495,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":770.97900000000004,"15min":770.97900000000004},"last":0.79700000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":51.109999999999999,"5min":14.760999999999999,"15min":18.811},"min":{"1min":0.39500000000000002,"5min":0.316,"15min":0.316},"max":{"1min":909.92600000000004,"5min":909.92600000000004,"15min":909.92600000000004},"last":0.76500000000000001},{"interface":"front","average":{"1min":27.942,"5min":13.455,"15min":4.8840000000000003},"min":{"1min":0.42199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":492.471,"5min":742.36199999999997,"15min":742.36199999999997},"last":0.65200000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":24.881,"15min":8.7479999999999993},"min":{"1min":0.49399999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":990.30100000000004,"15min":990.30100000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":44.832999999999998,"5min":9.4459999999999997,"15min":3.5779999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":840.41200000000003,"5min":840.41200000000003,"15min":840.41200000000003},"last":0.78400000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.8239999999999998,"15min":1.6739999999999999},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.755,"5min":323.27600000000001,"15min":323.27600000000001},"last":795.59199999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.622,"15min":0.621},"min":{"1min":0.52600000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.82299999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.74099999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.502},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.64300000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":39.576000000000001,"5min":8.4540000000000006,"15min":3.1909999999999998},"min":{"1min":0.36199999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":742.05999999999995,"5min":742.05999999999995,"15min":742.05999999999995},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":25.481999999999999,"5min":5.5170000000000003,"15min":2.2090000000000001},"min":{"1min":0.39900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":474.51900000000001,"5min":474.51900000000001,"15min":474.51900000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":37.012999999999998,"5min":17.600000000000001,"15min":6.2549999999999999},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":693.37699999999995,"5min":877.19000000000005,"15min":877.19000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Sep 30 00:29:29 2023","interfaces":[{"interface":"back","average":{"1min":34.536999999999999,"5min":13.4,"15min":4.8479999999999999},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":612.77599999999995,"5min":612.77599999999995,"15min":612.77599999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":7.9139999999999997,"15min":3.0489999999999999},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74099999999999999,"5min":733.37400000000002,"15min":733.37400000000002},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5720,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366120,"statfs":{"total":96636764160,"available":96630906880,"internally_reserved":0,"allocated":409600,"data_stored":184179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":3.8180000000000001,"5min":13.335000000000001,"15min":4.7249999999999996},"min":{"1min":0.33000000000000002,"5min":0.22,"15min":0.22},"max":{"1min":64.507000000000005,"5min":743.42399999999998,"15min":743.42399999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":9.3870000000000005,"15min":3.419},"min":{"1min":0.27800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.57799999999999996,"5min":845.18100000000004,"15min":845.18100000000004},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":13.286,"15min":17.943999999999999},"min":{"1min":0.33400000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.58899999999999997,"5min":668.12900000000002,"15min":668.12900000000002},"last":0.503},{"interface":"front","average":{"1min":0.48199999999999998,"5min":5.319,"15min":18.335000000000001},"min":{"1min":0.309,"5min":0.248,"15min":0.248},"max":{"1min":0.61399999999999999,"5min":585.68600000000004,"15min":585.68600000000004},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.7150000000000001,"15min":5.0979999999999999},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65200000000000002,"5min":148.685,"15min":148.685},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":5.8300000000000001,"15min":2.3239999999999998},"min":{"1min":0.37,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64500000000000002,"5min":532.24199999999996,"15min":532.24199999999996},"last":0.55200000000000005}]},{"osd":4,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":7.9459999999999997,"15min":26.808},"min":{"1min":0.46100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86899999999999999,"5min":501.096,"15min":501.096},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":4.7969999999999997,"15min":15.914999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82099999999999995,"5min":501.02300000000002,"15min":501.02300000000002},"last":0.59999999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":35.369999999999997,"5min":20.023,"15min":19.460999999999999},"min":{"1min":0.52600000000000002,"5min":0.435,"15min":0.435},"max":{"1min":694.34900000000005,"5min":962.88,"15min":962.88},"last":0.61499999999999999},{"interface":"front","average":{"1min":35.598999999999997,"5min":15.99,"15min":5.8120000000000003},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":699.37599999999998,"5min":794.64200000000005,"15min":794.64200000000005},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":7.7240000000000002,"15min":26.609999999999999},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84999999999999998,"5min":813.57399999999996,"15min":813.57399999999996},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.66600000000000004,"15min":0.73099999999999998},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:29:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65100000000000002,"15min":0.69499999999999995},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81299999999999994,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.67100000000000004,"15min":0.69399999999999995},"min":{"1min":0.51000000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88200000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6688,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94365152,"statfs":{"total":96636764160,"available":96629915648,"internally_reserved":0,"allocated":942080,"data_stored":713107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.30299999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.64900000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63500000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.436}]},{"osd":2,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.61299999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":8.8149999999999995,"15min":3.2309999999999999},"min":{"1min":0.27800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.67800000000000005,"5min":655.24400000000003,"15min":655.24400000000003},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":2.9449999999999998,"15min":1.4630000000000001},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60299999999999998,"5min":198.53800000000001,"15min":198.53800000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.29899999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65200000000000002,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":3.1080000000000001,"15min":9.7599999999999998},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92300000000000004,"5min":275.43200000000002,"15min":275.43200000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":16.745999999999999,"5min":3.8849999999999998,"15min":1.843},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":322.64999999999998,"5min":322.64999999999998,"15min":322.64999999999998},"last":0.67300000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":10.323,"15min":3.9889999999999999},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92500000000000004,"5min":772.57299999999998,"15min":772.57299999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64300000000000002,"15min":0.66200000000000003},"min":{"1min":0.47199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90800000000000003,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65000000000000002,"15min":0.64600000000000002},"min":{"1min":0.442,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97099999999999997,"5min":1.046,"15min":1.046},"last":0.72099999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:29:32 2023","interfaces":[{"interface":"back","average":{"1min":39.786999999999999,"5min":22.387,"15min":7.8959999999999999},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":822.63300000000004,"5min":990.16700000000003,"15min":990.16700000000003},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":2.5939999999999999,"15min":1.4390000000000001},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93799999999999994,"5min":198.173,"15min":198.173},"last":0.63700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":409600,"data_stored":184179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":29.780000000000001,"5min":18.908000000000001,"15min":24.202999999999999},"min":{"1min":0.34000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":499.13,"5min":586.52599999999995,"15min":586.52599999999995},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.59799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.44400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63600000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.379}]},{"osd":3,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.438,"15min":0.45000000000000001},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65600000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.501,"5min":9.2240000000000002,"15min":3.4119999999999999},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70199999999999996,"5min":655.45500000000004,"15min":655.45500000000004},"last":0.47599999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":8.4969999999999999,"15min":3.2869999999999999},"min":{"1min":0.49199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84999999999999998,"5min":708.62400000000002,"15min":708.62400000000002},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":7.3339999999999996,"15min":25.207000000000001},"min":{"1min":0.46800000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89500000000000002,"5min":536.221,"15min":536.221},"last":0.70699999999999996}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81899999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":20.416,"5min":13.345000000000001,"15min":4.9029999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":356.43000000000001,"5min":744.32600000000002,"15min":744.32600000000002},"last":0.76800000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.67100000000000004},"min":{"1min":0.44600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":9.8510000000000009,"15min":3.75},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90200000000000002,"5min":783.25300000000004,"15min":783.25300000000004},"last":0.69199999999999995}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":83.608999999999995,"5min":17.245999999999999,"15min":6.2199999999999998},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":678.30799999999999,"5min":678.30799999999999,"15min":678.30799999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.68799999999999994},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.67200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":409600,"data_stored":184179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.52800000000000002,"15min":0.81100000000000005},"min":{"1min":0.29499999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52900000000000003,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":9.6630000000000003,"15min":32.798000000000002},"min":{"1min":0.316,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.58699999999999997,"5min":733.85000000000002,"15min":733.85000000000002},"last":0.433}]},{"osd":1,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.30099999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63200000000000001,"5min":15.871,"15min":15.871},"last":0.40000000000000002},{"interface":"front","average":{"1min":88.762,"5min":18.300000000000001,"15min":7.0839999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":986.77999999999997,"5min":986.77999999999997,"15min":986.77999999999997},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.55700000000000005,"15min":0.89600000000000002},"min":{"1min":0.33200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.60799999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.504,"15min":0.622},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67100000000000004,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":4.0019999999999998,"15min":2.145},"min":{"1min":0.46200000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.77200000000000002,"5min":277.21499999999997,"15min":277.21499999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":3.931,"15min":1.8740000000000001},"min":{"1min":0.47799999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83799999999999997,"5min":309.69099999999997,"15min":309.69099999999997},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":29.396000000000001,"5min":14.071,"15min":5.4630000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":444.73000000000002,"5min":668.34500000000003,"15min":668.34500000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.76300000000000001,"15min":1.087},"min":{"1min":0.48099999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.88100000000000001,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.63100000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.68200000000000005,"15min":0.78400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84899999999999998,"5min":3.5289999999999999,"15min":3.5289999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.749,"15min":1.052},"min":{"1min":0.51200000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.76600000000000001,"5min":9.0950000000000006,"15min":9.0950000000000006},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:29:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.71299999999999997,"15min":0.86299999999999999},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90500000000000003,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":4.4299999999999997,"5min":1.5649999999999999,"15min":1.107},"min":{"1min":0.51100000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":68.840999999999994,"5min":68.840999999999994,"15min":68.840999999999994},"last":0.56499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":409600,"data_stored":184179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.39100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":3.8479999999999999,"15min":12.568},"min":{"1min":0.36799999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69599999999999995,"5min":360.43599999999998,"15min":360.43599999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.60699999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71799999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":11.301,"15min":4.1829999999999998},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83199999999999996,"5min":909.94100000000003,"15min":909.94100000000003},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.39400000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.749,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.62,"5min":12.66,"15min":4.657},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76000000000000001,"5min":963.85500000000002,"15min":963.85500000000002},"last":0.56000000000000005}]},{"osd":3,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":71.781000000000006,"5min":16.456,"15min":11.163},"min":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":680.40800000000002,"5min":680.40800000000002,"15min":680.40800000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":7.7240000000000002,"15min":3.02},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75800000000000001,"5min":565.77200000000005,"15min":565.77200000000005},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.503,"15min":0.501},"min":{"1min":0.373,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67300000000000004,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.37},{"interface":"front","average":{"1min":0.52800000000000002,"5min":4.7329999999999997,"15min":1.9179999999999999},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69799999999999995,"5min":463.30700000000002,"15min":463.30700000000002},"last":475.06599999999997}]},{"osd":6,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":6.1950000000000003,"15min":2.4169999999999998},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79200000000000004,"5min":453.32400000000001,"15min":453.32400000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":13.493,"15min":4.8789999999999996},"min":{"1min":0.45400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73199999999999998,"5min":596.84500000000003,"15min":596.84500000000003},"last":0.497}]},{"osd":7,"last update":"Sat Sep 30 00:29:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":5.6520000000000001,"15min":19.199999999999999},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80100000000000005,"5min":585.08000000000004,"15min":585.08000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":14.768000000000001,"15min":28.044},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.80900000000000005,"5min":794.49000000000001,"15min":794.49000000000001},"last":0.52700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5656,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94366184,"statfs":{"total":96636764160,"available":96630972416,"internally_reserved":0,"allocated":409600,"data_stored":184179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":17.231000000000002,"15min":6.1550000000000002},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.117,"5min":954.57299999999998,"15min":954.57299999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":5.3689999999999998,"15min":2.1989999999999998},"min":{"1min":0.46600000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":453.76100000000002,"15min":453.76100000000002},"last":0.65800000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":14.135999999999999,"15min":37.530999999999999},"min":{"1min":0.41099999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77100000000000002,"5min":973.71299999999997,"15min":973.71299999999997},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":8.8819999999999997,"15min":3.3660000000000001},"min":{"1min":0.42799999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.105,"5min":788.19899999999996,"15min":788.19899999999996},"last":0.79900000000000004}]},{"osd":2,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":2.915,"15min":1.383},"min":{"1min":0.42999999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.079,"5min":198.43000000000001,"15min":198.43000000000001},"last":0.78500000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60999999999999999,"15min":0.624},"min":{"1min":0.48799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.502,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77000000000000002,"5min":1.292,"15min":1.292},"last":0.70199999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":43.25,"5min":9.0299999999999994,"15min":3.3460000000000001},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.316},"max":{"1min":642.40800000000002,"5min":642.40800000000002,"15min":642.40800000000002},"last":0.625},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.495,"15min":0.52100000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.624,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69199999999999995,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":8.0060000000000002,"15min":27.940000000000001},"min":{"1min":0.41899999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67800000000000005,"5min":784.48299999999995,"15min":784.48299999999995},"last":0.71299999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:29:28 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52200000000000002,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":17.327000000000002,"15min":33.253999999999998},"min":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70399999999999996,"5min":853.28200000000004,"15min":853.28200000000004},"last":0.748}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:29:55.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:29:56.055 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:29:56.057 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:29:56.057 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:29:56.057 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:29:56.058 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.090 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:29:56.090 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:29:56.090 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.162 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:29:56.162 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:29:56.163 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.232 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:29:56.232 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:29:56.233 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.296 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:29:56.296 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T00:29:56.296 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.339 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T00:29:56.340 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T00:29:56.340 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.377 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T00:29:56.378 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T00:29:56.378 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.452 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T00:29:56.452 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T00:29:56.452 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.525 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T00:29:56.525 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:29:56.525 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T00:29:56.526 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.548 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T00:29:56.548 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:29:56.548 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-09-30T00:29:56.592 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:29:56.592 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:29:56.592 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:29:56.592 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.603 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:29:56.603 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T00:29:56.603 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.655 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T00:29:56.655 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T00:29:56.655 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:29:56.666 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T00:29:56.667 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:29:56.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:29:56.724 INFO:teuthology.orchestra.run.smithi031.stdout:1696033453.7879488 mgr.x (mgr.4104) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:29:56.724 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:29:56.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:29:56.794 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:29:56.862 INFO:teuthology.orchestra.run.smithi031.stdout:1696033453.7879488 mgr.x (mgr.4104) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:29:56.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:29:56.863 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:29:57.112 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:29:57.112 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:29:57.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:29:57.230 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:29:57.299 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:29:57.299 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:29:57.365 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:29:57.365 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T00:29:57.594 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:29:57.594 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T00:29:57.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:29:57.678 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T00:29:57.746 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:29:57.747 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T00:29:57.816 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:29:57.817 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/data/mon.a.tgz 2023-09-30T00:29:57.817 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-09-30T00:29:57.833 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.c2IEGVjkow 2023-09-30T00:29:57.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.c2IEGVjkow 2023-09-30T00:29:57.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.c2IEGVjkow 2023-09-30T00:29:58.071 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.c2IEGVjkow is 292KB 2023-09-30T00:29:58.166 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.c2IEGVjkow 2023-09-30T00:29:58.180 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/data/mon.c.tgz 2023-09-30T00:29:58.181 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-09-30T00:29:58.234 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.DHPiZWxD7g 2023-09-30T00:29:58.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.DHPiZWxD7g 2023-09-30T00:29:58.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.DHPiZWxD7g 2023-09-30T00:29:58.430 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.DHPiZWxD7g is 293KB 2023-09-30T00:29:58.523 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.DHPiZWxD7g 2023-09-30T00:29:58.537 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/data/mon.b.tgz 2023-09-30T00:29:58.537 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2023-09-30T00:29:58.552 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.S8joWfDoMQ 2023-09-30T00:29:58.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.S8joWfDoMQ 2023-09-30T00:29:58.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.S8joWfDoMQ 2023-09-30T00:29:58.779 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.S8joWfDoMQ is 293KB 2023-09-30T00:29:58.872 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.S8joWfDoMQ 2023-09-30T00:29:58.887 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:29:58.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:29:58.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:29:58.950 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:29:58.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:29:58.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:29:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:29:59.015 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:29:59.022 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:29:59.023 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/remote/smithi031/crash 2023-09-30T00:29:59.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:29:59.051 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/remote/smithi146/crash 2023-09-30T00:29:59.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:29:59.087 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:29:59.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:29:59.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:30:00.113 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:30:00.114 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/remote/smithi031/log 2023-09-30T00:30:00.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:30:00.405 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/remote/smithi146/log 2023-09-30T00:30:00.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:30:00.553 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:30:00.627 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ecd606989cf9401fae6bd288a0c37a18 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:30:00.630 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:30:00.652 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:30:00.667 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:30:00.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:30:00.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:30:00.749 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:30:00.757 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:30:00.757 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:30:00.760 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:30:00.771 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-09-30T00:30:00.773 INFO:teuthology.orchestra.run.smithi146.stderr:bash: ntpq: command not found 2023-09-30T00:30:00.775 INFO:teuthology.orchestra.run.smithi031.stdout:210 Number of sources = 4 2023-09-30T00:30:00.775 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:30:00.775 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-09-30T00:30:00.775 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 -353us[ -353us] +/- 68ms 2023-09-30T00:30:00.775 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -360us[ -360us] +/- 53ms 2023-09-30T00:30:00.775 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv03.front.sepia.ceph.com 3 6 377 64 +949us[ +941us] +/- 73ms 2023-09-30T00:30:00.776 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:30:00.776 INFO:teuthology.orchestra.run.smithi146.stdout:210 Number of sources = 4 2023-09-30T00:30:00.777 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:30:00.777 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2023-09-30T00:30:00.777 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 -367us[ -347us] +/- 68ms 2023-09-30T00:30:00.777 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 0 -341us[ -341us] +/- 54ms 2023-09-30T00:30:00.777 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv03.front.sepia.ceph.com 3 6 377 63 +941us[ +961us] +/- 73ms 2023-09-30T00:30:00.777 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:30:00.778 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:30:00.786 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:30:00.786 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:30:00.795 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:30:00.910 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:30:00.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:30:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:30:01.028 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:30:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:30:01.057 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:30:01.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:30:01.078 INFO:teuthology.task.internal:Duration was 1151.154832 seconds 2023-09-30T00:30:01.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:30:01.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:30:01.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:30:01.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:30:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:30:01.142 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:30:01.427 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:30:01.428 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-09-30T00:30:01.428 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:30:01.449 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2023-09-30T00:30:01.449 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:30:01.468 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:30:01.468 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:30:01.492 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:30:01.545 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:30:01.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:30:01.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:30:01.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-09-30T00:30:01.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:30:01.669 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:30:01.669 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:30:01.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:30:01.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:30:01.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:30:01.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:30:01.760 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-09-30T00:30:01.783 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2023-09-30T00:30:01.797 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:30:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:30:01.828 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:30:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:30:01.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:30:01.861 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:30:01.861 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/remote/smithi031 2023-09-30T00:30:01.862 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:30:01.908 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696/remote/smithi146 2023-09-30T00:30:01.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:30:01.945 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:30:01.945 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:30:01.947 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:30:01.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:30:02.005 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:30:02.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:30:02.015 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:30:02.016 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:30:02.018 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:30:02.030 INFO:teuthology.orchestra.run.smithi031.stdout: 262176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:30 /home/ubuntu/cephtest 2023-09-30T00:30:02.047 INFO:teuthology.orchestra.run.smithi146.stdout: 262183 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:30 /home/ubuntu/cephtest 2023-09-30T00:30:02.049 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:30:02.058 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:30:02.077 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:30:02.105 DEBUG:teuthology.nuke:shortname: smithi031 2023-09-30T00:30:02.105 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:30:02.130 DEBUG:teuthology.nuke:shortname: smithi146 2023-09-30T00:30:02.130 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:30:02.134 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:03:42.587198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbLSWpInlss1NhwAmRhZO8PmH0HZUnnVKkzJwG9jwzE9fx6pDf+J6C0HPNEsaoc1Uc10f1MWySqmXKWh7mnFgE='} 2023-09-30T00:30:02.158 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406696', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:03:42.588725', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCw9suGSXpcUQH2x096mCSKwsSx8zzV4kALqY6WlEzmOhUXQvknAtq3ZjgAPq4f1J85RnSx+ZyY6CVcP57h+Fsw='} 2023-09-30T00:30:02.186 INFO:teuthology.orchestra.console.smithi031:Power off 2023-09-30T00:30:02.187 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:30:02.212 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-09-30T00:30:02.213 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:02.230 INFO:teuthology.orchestra.console.smithi146:Power off 2023-09-30T00:30:02.230 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:30:02.239 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-09-30T00:30:02.253 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2023-09-30T00:30:02.253 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:02.276 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2023-09-30T00:30:06.240 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:06.277 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:06.352 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-09-30T00:30:06.386 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2023-09-30T00:30:10.354 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:10.388 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:10.465 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-09-30T00:30:10.497 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2023-09-30T00:30:10.566 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-09-30T00:30:14.498 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:30:14.607 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2023-09-30T00:30:14.708 INFO:teuthology.orchestra.console.smithi146:Power off completed 2023-09-30T00:30:14.813 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} duration: 1151.1548323631287 failure_reason: '"1696033453.7879488 mgr.x (mgr.4104) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ecd606989cf9401fae6bd288a0c37a18 status: fail success: false 2023-09-30T00:30:14.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:30:14.896 INFO:teuthology.run:FAIL