2023-09-30T00:21:16.136 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:21:16.136 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725 --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --verbose --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/orig.config.yaml 2023-09-30T00:21:16.180 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:21:16.263 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} email: yweinste@redhat.com first_in_suite: false job_id: '7406725' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon election default strategy: 1 ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd inject bad map crc probability: 0.1 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd max scrubs: 2 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd sloppy crc: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - must scrub before tier agent can activate sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache readproxy - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache target_max_objects 250 - rados: clients: - client.0 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pool_snaps: true pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:21:16.263 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:21:16.264 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:21:16.264 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:21:16.265 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:21:16.285 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:21:16.285 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:21:16.285 INFO:teuthology.packaging:ref: None 2023-09-30T00:21:16.286 INFO:teuthology.packaging:tag: None 2023-09-30T00:21:16.286 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:21:16.286 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:21:16.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:21:16.425 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:21:16.426 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:21:16.432 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:21:16.432 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:21:16.437 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:21:16.453 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:21:16.458 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:21:16.483 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:14:35.498712', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T00:21:16.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:14:35.500144', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T00:21:16.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:21:16.515 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T00:21:16.515 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T00:21:16.515 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:21:16.581 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa5198400d0>, signals=[15]) 2023-09-30T00:21:16.581 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:21:16.588 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:21:16.588 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:21:16.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:21:16.639 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:21:16.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:21:16.687 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:21:16.698 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2023-09-30T00:21:16.718 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2023-09-30T00:21:16.718 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2023-09-30T00:21:16.766 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="20.04" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=focal 2023-09-30T00:21:16.767 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=focal 2023-09-30T00:21:16.768 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-09-30T00:21:16.788 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2023-09-30T00:21:16.794 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2023-09-30T00:21:16.794 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2023-09-30T00:21:16.843 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Ubuntu" 2023-09-30T00:21:16.843 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T00:21:16.843 INFO:teuthology.orchestra.run.smithi157.stdout:ID=ubuntu 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE=debian 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="20.04" 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_CODENAME=focal 2023-09-30T00:21:16.844 INFO:teuthology.orchestra.run.smithi157.stdout:UBUNTU_CODENAME=focal 2023-09-30T00:21:16.844 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2023-09-30T00:21:16.871 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:21:16.880 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:21:16.885 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:21:16.885 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:21:16.887 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:21:16.893 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:21:16.898 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:21:16.898 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:21:16.901 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:21:16.951 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:21:17.003 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:21:17.016 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:21:17.017 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:21:17.017 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:21:17.017 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:21:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:21:17.023 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-09-30T00:21:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:21:17.024 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-09-30T00:21:17.030 INFO:teuthology.orchestra.run.smithi110.stdout:5.4.0-124-generic 2023-09-30T00:21:17.030 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.4.0-124-generic 2023-09-30T00:21:17.030 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2023-09-30T00:21:17.031 INFO:teuthology.orchestra.run.smithi157.stdout:5.4.0-124-generic 2023-09-30T00:21:17.031 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.4.0-124-generic 2023-09-30T00:21:17.032 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2023-09-30T00:21:17.131 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-09-30T00:21:17.139 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2023-09-30T00:21:17.414 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:21:17.422 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T00:21:17.422 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:21:17.423 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T00:21:17.472 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T00:21:17.479 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T00:21:17.816 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T00:21:17.826 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T00:21:17.907 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T00:21:17.909 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T00:21:18.042 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T00:21:18.048 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T00:21:18.394 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T00:21:18.395 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T00:21:18.409 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T00:21:18.418 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T00:21:18.418 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T00:21:18.419 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T00:21:18.430 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T00:21:18.445 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T00:21:18.454 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T00:21:18.454 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T00:21:18.455 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T00:21:18.493 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T00:21:18.500 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T00:21:18.500 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T00:21:18.512 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T00:21:18.515 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T00:21:18.529 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T00:21:18.529 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T00:21:18.536 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T00:21:18.536 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T00:21:18.538 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T00:21:18.539 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T00:21:18.539 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T00:21:18.550 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T00:21:18.608 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T00:21:18.613 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T00:21:18.619 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T00:21:18.621 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T00:21:18.622 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T00:21:18.623 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T00:21:18.697 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T00:21:18.698 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T00:21:18.704 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T00:21:18.734 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T00:21:18.783 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T00:21:18.869 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T00:21:19.117 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T00:21:19.123 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T00:21:19.131 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T00:21:19.133 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T00:21:19.135 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T00:21:19.151 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T00:21:19.157 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T00:21:19.159 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T00:21:19.170 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T00:21:19.170 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T00:21:19.415 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T00:21:19.455 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T00:21:19.457 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T00:21:19.467 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T00:21:19.468 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T00:21:19.469 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T00:21:19.471 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T00:21:19.471 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T00:21:19.471 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T00:21:19.473 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T00:21:19.499 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T00:21:19.566 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T00:21:19.587 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T00:21:19.589 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T00:21:19.589 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T00:21:19.591 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T00:21:19.592 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T00:21:19.592 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T00:21:19.595 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T00:21:19.597 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T00:21:19.663 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T00:21:19.761 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T00:21:19.846 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T00:21:19.868 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T00:21:19.945 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T00:21:19.967 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T00:21:22.254 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 17.4 MB in 3s (5,651 kB/s) 2023-09-30T00:21:22.280 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 17.4 MB in 3s (5,698 kB/s) 2023-09-30T00:21:23.674 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-09-30T00:21:23.693 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2023-09-30T00:21:23.700 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T00:21:23.721 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T00:21:23.778 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-09-30T00:21:23.802 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2023-09-30T00:21:23.965 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-09-30T00:21:23.966 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-09-30T00:21:24.005 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2023-09-30T00:21:24.005 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2023-09-30T00:21:24.137 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-09-30T00:21:24.137 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T00:21:24.137 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T00:21:24.137 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T00:21:24.138 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-09-30T00:21:24.138 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T00:21:24.138 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2023-09-30T00:21:24.139 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2023-09-30T00:21:24.187 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2023-09-30T00:21:24.187 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T00:21:24.187 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T00:21:24.187 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T00:21:24.188 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2023-09-30T00:21:24.188 INFO:teuthology.orchestra.run.smithi157.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T00:21:24.188 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2023-09-30T00:21:24.188 INFO:teuthology.orchestra.run.smithi157.stdout: thermald 2023-09-30T00:21:24.202 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-09-30T00:21:24.202 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T00:21:24.202 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T00:21:24.203 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T00:21:24.204 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-09-30T00:21:24.204 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T00:21:24.248 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2023-09-30T00:21:24.248 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T00:21:24.248 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T00:21:24.248 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T00:21:24.249 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages will be upgraded: 2023-09-30T00:21:24.250 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T00:21:24.255 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T00:21:24.256 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 77.1 MB of archives. 2023-09-30T00:21:24.256 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T00:21:24.256 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T00:21:24.431 INFO:teuthology.orchestra.run.smithi157.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T00:21:24.431 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 77.1 MB of archives. 2023-09-30T00:21:24.431 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T00:21:24.431 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T00:21:24.590 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T00:21:24.682 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T00:21:25.146 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T00:21:25.146 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T00:21:25.147 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T00:21:25.237 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T00:21:25.247 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T00:21:25.576 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T00:21:25.892 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T00:21:25.963 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 77.1 MB in 1s (74.3 MB/s) 2023-09-30T00:21:26.163 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T00:21:27.206 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T00:21:27.206 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T00:21:27.206 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T00:21:27.488 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T00:21:27.516 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T00:21:27.934 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T00:21:27.939 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:28.035 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:28.223 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 77.1 MB in 3s (23.6 MB/s) 2023-09-30T00:21:28.427 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T00:21:30.010 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T00:21:30.024 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:30.122 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:30.166 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T00:21:30.171 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:30.266 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:31.221 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T00:21:31.234 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:31.271 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:32.158 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T00:21:32.172 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:32.289 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:33.387 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T00:21:33.400 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:33.436 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:36.070 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:21:36.195 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:21:36.563 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:21:36.691 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:21:37.044 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T00:21:37.059 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T00:21:37.102 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T00:21:38.160 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:21:38.300 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:21:38.620 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:21:38.729 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:21:39.065 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T00:21:39.079 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T00:21:39.123 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T00:21:42.376 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T00:21:42.398 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:42.442 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:45.090 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:21:45.134 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T00:21:45.156 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:21:45.201 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:45.243 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:21:45.568 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:46.856 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T00:21:46.957 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:47.916 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:21:48.051 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:21:48.091 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:21:48.091 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T00:21:48.243 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:48.377 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:48.384 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:49.545 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T00:21:49.624 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T00:21:49.696 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T00:21:49.750 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:49.814 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T00:21:49.915 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:49.979 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T00:21:49.979 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:21:51.339 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:21:51.340 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T00:21:51.477 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:51.603 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:52.780 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T00:21:52.922 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T00:21:53.048 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T00:21:53.175 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:21:53.241 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T00:21:53.241 INFO:teuthology.orchestra.run.smithi157.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:22:12.027 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T00:22:12.027 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2023-09-30T00:22:12.045 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T00:22:12.102 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2023-09-30T00:22:12.472 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:22:12.503 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:22:12.974 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T00:22:12.986 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T00:22:15.431 INFO:teuthology.orchestra.run.smithi110.stdout:done 2023-09-30T00:22:15.659 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T00:22:15.659 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub' 2023-09-30T00:22:15.677 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T00:22:15.687 INFO:teuthology.orchestra.run.smithi157.stdout:Generating grub configuration file ... 2023-09-30T00:22:15.827 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2023-09-30T00:22:15.875 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 20 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2023-09-30T00:22:15.876 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.4.0.163.160 2023-09-30T00:22:15.877 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T00:22:15.877 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T00:22:15.877 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2023-09-30T00:22:15.877 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2023-09-30T00:22:15.877 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2023-09-30T00:22:15.877 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2023-09-30T00:22:15.877 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T00:22:15.878 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:22:15.878 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:22:15.878 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:22:15.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:22:15.878 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2023-09-30T00:22:15.997 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-09-30T00:22:16.024 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:22:16.042 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:22:16.134 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T00:22:16.140 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:22:16.160 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T00:22:16.180 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T00:22:16.437 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T00:22:16.447 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T00:22:17.847 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-09-30T00:22:17.871 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T00:22:17.940 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-09-30T00:22:18.134 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-09-30T00:22:18.135 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-09-30T00:22:18.341 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T00:22:18.342 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T00:22:18.344 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2023-09-30T00:22:18.365 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2023-09-30T00:22:18.365 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2023-09-30T00:22:18.365 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2023-09-30T00:22:18.365 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2023-09-30T00:22:18.365 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 20 2023-09-30T00:22:18.365 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.4.0.163.160 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2023-09-30T00:22:18.366 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2023-09-30T00:22:18.366 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-09-30T00:22:18.371 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.Ba3E7cE2qD 2023-09-30T00:22:18.371 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Ba3E7cE2qD 2023-09-30T00:22:18.385 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.Ba3E7cE2qD 2023-09-30T00:22:18.473 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.Ba3E7cE2qD is 9KB 2023-09-30T00:22:18.488 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.Ba3E7cE2qD 2023-09-30T00:22:18.494 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T00:22:18.552 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:22:18.552 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T00:22:18.552 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T00:22:18.595 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T00:22:18.595 DEBUG:teuthology.orchestra.run.smithi110:> sudo update-grub 2023-09-30T00:22:19.160 INFO:teuthology.orchestra.run.smithi157.stdout:done 2023-09-30T00:22:19.565 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2023-09-30T00:22:19.613 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2023-09-30T00:22:19.613 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 20 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2023-09-30T00:22:19.614 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.4.0.163.160 2023-09-30T00:22:19.615 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T00:22:19.615 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T00:22:19.615 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2023-09-30T00:22:19.615 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2023-09-30T00:22:19.615 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2023-09-30T00:22:19.615 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2023-09-30T00:22:19.616 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T00:22:19.617 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:22:19.617 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:22:19.617 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T00:22:19.617 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:22:19.617 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2023-09-30T00:22:19.692 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2023-09-30T00:22:19.827 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T00:22:19.902 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2023-09-30T00:22:19.910 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T00:22:19.940 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:22:19.966 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2023-09-30T00:22:20.022 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T00:22:20.103 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T00:22:20.250 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:22:20.262 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:22:20.687 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T00:22:20.707 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T00:22:21.558 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2023-09-30T00:22:21.584 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T00:22:21.653 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2023-09-30T00:22:21.853 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2023-09-30T00:22:21.854 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2023-09-30T00:22:22.072 INFO:teuthology.orchestra.run.smithi157.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T00:22:22.072 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T00:22:22.074 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2023-09-30T00:22:22.088 INFO:teuthology.orchestra.run.smithi110.stderr:done 2023-09-30T00:22:22.089 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2023-09-30T00:22:22.091 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2023-09-30T00:22:22.091 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2023-09-30T00:22:22.091 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2023-09-30T00:22:22.091 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2023-09-30T00:22:22.091 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 20 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.4.0.163.160 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2023-09-30T00:22:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2023-09-30T00:22:22.092 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-09-30T00:22:22.141 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.tt2eY895fE 2023-09-30T00:22:22.141 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tt2eY895fE 2023-09-30T00:22:22.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.tt2eY895fE 2023-09-30T00:22:22.195 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.tt2eY895fE is 9KB 2023-09-30T00:22:22.208 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.tt2eY895fE 2023-09-30T00:22:22.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T00:22:22.268 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:22:22.268 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T00:22:22.268 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T00:22:22.296 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T00:22:22.296 DEBUG:teuthology.orchestra.run.smithi157:> sudo update-grub 2023-09-30T00:22:23.597 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub' 2023-09-30T00:22:23.600 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T00:22:23.612 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2023-09-30T00:22:23.883 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:22:23.894 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:22:24.283 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T00:22:24.295 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T00:22:25.969 INFO:teuthology.orchestra.run.smithi157.stderr:done 2023-09-30T00:22:25.970 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2023-09-30T00:22:52.106 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:22:52.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T00:22:52.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:22:55.977 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T00:22:55.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:22:55.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:23:10.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-09-30T00:23:14.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:23:19.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T00:23:19.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:23:22.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-09-30T00:23:23.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:23:23.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:23:26.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:23:34.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T00:23:34.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:23:38.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:23:38.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:23:41.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:23:56.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:23:56.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:24:06.232 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-09-30T00:24:06.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T00:24:06.788 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-163-generic"... 2023-09-30T00:24:06.788 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-09-30T00:24:06.794 INFO:teuthology.orchestra.run.smithi110.stdout:5.4.0-163-generic 2023-09-30T00:24:06.794 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T00:24:06.794 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:24:06.795 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T00:24:07.796 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:24:07.796 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:24:07.882 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2023-09-30T00:24:07.912 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:14.970 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-09-30T00:24:32.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:24:32.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:24:33.178 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-09-30T00:24:33.709 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:24:33.709 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-163-generic"... 2023-09-30T00:24:33.710 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2023-09-30T00:24:33.716 INFO:teuthology.orchestra.run.smithi157.stdout:5.4.0-163-generic 2023-09-30T00:24:33.716 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T00:24:33.716 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:24:33.716 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T00:24:34.717 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T00:24:34.718 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:24:34.806 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2023-09-30T00:24:34.835 DEBUG:teuthology.parallel:result is None 2023-09-30T00:24:34.835 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:24:34.844 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:24:34.844 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:24:34.847 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:24:34.854 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:24:34.860 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:24:34.866 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:24:34.867 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:24:34.896 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:24:34.931 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:24:34.938 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:24:34.938 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:24:34.940 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:24:34.963 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:24:34.973 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:24:34.986 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:24:34.996 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:24:34.998 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:24:35.004 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:24:35.005 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:24:35.007 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:24:35.023 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:24:35.032 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:24:35.032 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:24:35.064 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:24:35.071 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:24:35.079 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:24:35.086 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:24:35.086 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:24:35.109 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:24:35.119 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:24:35.126 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:24:35.126 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:24:35.147 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2023-09-30T00:24:35.152 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2023-09-30T00:24:35.197 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:24:35.206 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:24:35.206 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:24:35.215 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:24:35.222 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2023-09-30T00:24:35.222 DEBUG:teuthology.task.selinux:Excluding smithi157: OS 'ubuntu' does not support SELinux 2023-09-30T00:24:35.223 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:24:35.223 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-30T00:24:35.223 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:24:35.223 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:24:35.231 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:24:35.231 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:24:35.246 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:24:35.248 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:30:15.743 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2023-09-30T00:30:15.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T00:30:15.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:30:15.796 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-09-30T00:30:15.916 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T00:30:15.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:30:15.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:30:15.964 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-09-30T00:30:16.088 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-09-30T00:30:16.088 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:30:16.101 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:30:16.101 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:30:16.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:30:16.103 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:30:16.104 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:30:16.134 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T00:30:16.134 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Command line: ntpd -gq 2023-09-30T00:30:16.135 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: proto: precision = 0.078 usec (-24) 2023-09-30T00:30:16.136 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T00:30:16.136 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T00:30:16.137 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listen normally on 4 lo [::1]:123 2023-09-30T00:30:16.138 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2023-09-30T00:30:16.139 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:16 ntpd[22726]: Listening on routing socket on fd #22 for interface updates 2023-09-30T00:30:16.139 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T00:30:16.140 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Command line: ntpd -gq 2023-09-30T00:30:16.140 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: proto: precision = 0.100 usec (-23) 2023-09-30T00:30:16.140 INFO:teuthology.orchestra.run.smithi157.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T00:30:16.140 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T00:30:16.141 INFO:teuthology.orchestra.run.smithi157.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listen normally on 3 enp3s0f1 172.21.15.157:123 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listen normally on 4 lo [::1]:123 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea1%5]:123 2023-09-30T00:30:16.142 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:16 ntpd[22739]: Listening on routing socket on fd #22 for interface updates 2023-09-30T00:30:17.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:17.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:17 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:17.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:17 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:17 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:17 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:17.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:17 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:17.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:17 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:17.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:17 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:19.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:19.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:19 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:19.142 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.142 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.142 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:19.143 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:19 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:20.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:20 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:20.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:20 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:20.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:20 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:20.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:20 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:20.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:20 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:20.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:20 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:21.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:21 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:21.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:21.141 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:21 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:22.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:22 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:22.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:22 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:22.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:22 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:22.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:22 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:22.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:22 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:22.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:22 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.136 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:23.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:23.137 INFO:teuthology.orchestra.run.smithi110.stdout:30 Sep 00:30:23 ntpd[22726]: ntpd: time slew +0.001278 s 2023-09-30T00:30:23.137 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time slew +0.001278s 2023-09-30T00:30:23.137 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T00:30:23.138 INFO:teuthology.orchestra.run.smithi110.stderr:30 Sep 00:30:23 ntpd[22726]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T00:30:23.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.139 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:30:23.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:30:23.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:30:23.140 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:30:23.140 INFO:teuthology.orchestra.run.smithi157.stdout:30 Sep 00:30:23 ntpd[22739]: ntpd: time slew -0.000631 s 2023-09-30T00:30:23.140 INFO:teuthology.orchestra.run.smithi157.stdout:ntpd: time slew -0.000631s 2023-09-30T00:30:23.141 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T00:30:23.141 INFO:teuthology.orchestra.run.smithi157.stderr:30 Sep 00:30:23 ntpd[22739]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T00:30:23.195 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T00:30:23.195 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2023-09-30T00:30:23.195 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.195 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.195 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.195 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.199 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T00:30:23.199 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2023-09-30T00:30:23.199 INFO:teuthology.orchestra.run.smithi157.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.200 INFO:teuthology.orchestra.run.smithi157.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.200 INFO:teuthology.orchestra.run.smithi157.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.200 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:30:23.201 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:30:23.209 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:30:23.209 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:30:23.209 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:30:23.209 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:30:23.214 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:30:23.214 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:30:23.215 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2023-09-30T00:30:23.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-key list | grep Ceph 2023-09-30T00:30:23.340 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T00:30:23.347 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T00:30:23.398 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T00:30:23.398 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T00:30:23.399 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T00:30:23.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:30:23.407 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T00:30:23.408 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T00:30:23.408 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T00:30:23.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:30:23.547 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T00:30:23.547 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:30:23.617 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T00:30:23.618 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:30:23.656 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:30:23.656 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T00:30:23.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2023-09-30T00:30:23.787 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:30:23.787 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T00:30:23.804 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-09-30T00:30:23.982 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T00:30:23.988 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T00:30:23.997 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:30:23.997 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T00:30:24.019 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T00:30:24.057 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T00:30:24.067 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T00:30:24.093 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T00:30:24.103 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:30:24.104 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T00:30:24.141 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T00:30:24.149 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T00:30:24.155 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T00:30:24.179 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T00:30:24.185 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T00:30:24.216 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T00:30:24.276 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T00:30:24.276 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T00:30:24.637 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 28.4 kB in 1s (37.9 kB/s) 2023-09-30T00:30:24.647 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 28.4 kB in 1s (44.8 kB/s) 2023-09-30T00:30:25.666 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2023-09-30T00:30:25.676 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-09-30T00:30:25.689 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:30:25.702 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:30:25.756 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2023-09-30T00:30:25.774 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-09-30T00:30:25.955 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2023-09-30T00:30:25.956 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2023-09-30T00:30:25.971 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-09-30T00:30:25.972 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-09-30T00:30:26.136 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T00:30:26.137 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T00:30:26.137 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T00:30:26.138 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2023-09-30T00:30:26.138 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T00:30:26.138 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T00:30:26.138 INFO:teuthology.orchestra.run.smithi157.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T00:30:26.138 INFO:teuthology.orchestra.run.smithi157.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T00:30:26.138 INFO:teuthology.orchestra.run.smithi157.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T00:30:26.139 INFO:teuthology.orchestra.run.smithi157.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T00:30:26.140 INFO:teuthology.orchestra.run.smithi157.stdout: socat xmlstarlet 2023-09-30T00:30:26.141 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2023-09-30T00:30:26.141 INFO:teuthology.orchestra.run.smithi157.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T00:30:26.141 INFO:teuthology.orchestra.run.smithi157.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T00:30:26.141 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2023-09-30T00:30:26.142 INFO:teuthology.orchestra.run.smithi157.stdout: btrfs-tools 2023-09-30T00:30:26.143 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T00:30:26.143 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T00:30:26.143 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T00:30:26.143 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T00:30:26.144 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: socat xmlstarlet 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-09-30T00:30:26.145 INFO:teuthology.orchestra.run.smithi110.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2023-09-30T00:30:26.146 INFO:teuthology.orchestra.run.smithi110.stdout: btrfs-tools 2023-09-30T00:30:26.254 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2023-09-30T00:30:26.254 INFO:teuthology.orchestra.run.smithi157.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T00:30:26.255 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T00:30:26.255 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T00:30:26.255 INFO:teuthology.orchestra.run.smithi157.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T00:30:26.255 INFO:teuthology.orchestra.run.smithi157.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T00:30:26.255 INFO:teuthology.orchestra.run.smithi157.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T00:30:26.255 INFO:teuthology.orchestra.run.smithi157.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T00:30:26.256 INFO:teuthology.orchestra.run.smithi157.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T00:30:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages will be upgraded: 2023-09-30T00:30:26.258 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 librbd1 2023-09-30T00:30:26.259 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-09-30T00:30:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T00:30:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T00:30:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T00:30:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T00:30:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T00:30:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T00:30:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T00:30:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T00:30:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T00:30:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T00:30:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T00:30:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T00:30:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T00:30:26.262 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-09-30T00:30:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 librbd1 2023-09-30T00:30:26.323 INFO:teuthology.orchestra.run.smithi110.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T00:30:26.323 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 209 MB of archives. 2023-09-30T00:30:26.323 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T00:30:26.323 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T00:30:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T00:30:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 209 MB of archives. 2023-09-30T00:30:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T00:30:26.327 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T00:30:26.405 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T00:30:26.406 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T00:30:26.415 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T00:30:26.418 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T00:30:26.440 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T00:30:26.441 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T00:30:26.444 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T00:30:26.445 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T00:30:26.449 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T00:30:26.450 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T00:30:26.478 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T00:30:26.480 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T00:30:26.482 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T00:30:26.488 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T00:30:26.490 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T00:30:26.490 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T00:30:26.493 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T00:30:26.493 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T00:30:26.493 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T00:30:26.495 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T00:30:26.498 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T00:30:26.512 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T00:30:26.512 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T00:30:26.513 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T00:30:26.514 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T00:30:26.514 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T00:30:26.514 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T00:30:26.515 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T00:30:26.516 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T00:30:26.517 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T00:30:26.532 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T00:30:26.532 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T00:30:26.536 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T00:30:26.538 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T00:30:26.539 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T00:30:26.539 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T00:30:26.548 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T00:30:26.548 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T00:30:26.549 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T00:30:26.551 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T00:30:26.552 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T00:30:26.552 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T00:30:26.565 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T00:30:26.566 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T00:30:26.568 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T00:30:26.568 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T00:30:26.573 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T00:30:26.582 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T00:30:26.645 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T00:30:26.646 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T00:30:26.659 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T00:30:26.663 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T00:30:26.667 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T00:30:26.674 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T00:30:26.685 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T00:30:26.688 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T00:30:26.689 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T00:30:26.691 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T00:30:26.692 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T00:30:26.692 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T00:30:26.694 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T00:30:26.694 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T00:30:26.730 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T00:30:26.740 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T00:30:26.741 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T00:30:26.741 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T00:30:26.743 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T00:30:26.748 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T00:30:26.748 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T00:30:26.748 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T00:30:26.749 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T00:30:26.749 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T00:30:26.749 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T00:30:26.750 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T00:30:26.751 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T00:30:26.751 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T00:30:26.765 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T00:30:26.767 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T00:30:26.782 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T00:30:26.785 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T00:30:26.786 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T00:30:26.791 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T00:30:26.792 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T00:30:26.793 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T00:30:26.793 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T00:30:26.795 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T00:30:26.796 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T00:30:26.798 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T00:30:26.800 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T00:30:26.808 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T00:30:26.810 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T00:30:26.846 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T00:30:26.847 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T00:30:26.847 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T00:30:26.848 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T00:30:26.848 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T00:30:26.848 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T00:30:26.853 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T00:30:26.860 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T00:30:26.860 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T00:30:26.861 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T00:30:26.863 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T00:30:26.880 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T00:30:26.881 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T00:30:26.882 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T00:30:26.882 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T00:30:26.885 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T00:30:26.885 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T00:30:26.925 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T00:30:26.925 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T00:30:26.926 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T00:30:26.927 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T00:30:26.929 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T00:30:26.933 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T00:30:26.934 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T00:30:26.937 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T00:30:26.938 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T00:30:26.945 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T00:30:26.946 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T00:30:26.975 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T00:30:26.976 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T00:30:27.228 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T00:30:27.266 INFO:teuthology.orchestra.run.smithi157.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T00:30:27.268 INFO:teuthology.orchestra.run.smithi157.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T00:30:27.363 INFO:teuthology.orchestra.run.smithi157.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T00:30:27.363 INFO:teuthology.orchestra.run.smithi157.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T00:30:27.364 INFO:teuthology.orchestra.run.smithi157.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T00:30:27.364 INFO:teuthology.orchestra.run.smithi157.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T00:30:27.364 INFO:teuthology.orchestra.run.smithi157.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T00:30:27.365 INFO:teuthology.orchestra.run.smithi157.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T00:30:27.365 INFO:teuthology.orchestra.run.smithi157.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T00:30:27.366 INFO:teuthology.orchestra.run.smithi157.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T00:30:27.366 INFO:teuthology.orchestra.run.smithi157.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T00:30:27.408 INFO:teuthology.orchestra.run.smithi157.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T00:30:27.413 INFO:teuthology.orchestra.run.smithi157.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T00:30:27.469 INFO:teuthology.orchestra.run.smithi157.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T00:30:27.471 INFO:teuthology.orchestra.run.smithi157.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T00:30:27.505 INFO:teuthology.orchestra.run.smithi157.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T00:30:27.508 INFO:teuthology.orchestra.run.smithi157.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T00:30:27.511 INFO:teuthology.orchestra.run.smithi157.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T00:30:27.546 INFO:teuthology.orchestra.run.smithi157.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T00:30:27.548 INFO:teuthology.orchestra.run.smithi157.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T00:30:27.632 INFO:teuthology.orchestra.run.smithi157.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T00:30:27.632 INFO:teuthology.orchestra.run.smithi157.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T00:30:27.632 INFO:teuthology.orchestra.run.smithi157.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T00:30:27.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T00:30:27.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T00:30:27.635 INFO:teuthology.orchestra.run.smithi157.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T00:30:27.653 INFO:teuthology.orchestra.run.smithi157.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T00:30:27.653 INFO:teuthology.orchestra.run.smithi157.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T00:30:27.653 INFO:teuthology.orchestra.run.smithi157.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T00:30:27.653 INFO:teuthology.orchestra.run.smithi157.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T00:30:27.673 INFO:teuthology.orchestra.run.smithi157.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T00:30:27.674 INFO:teuthology.orchestra.run.smithi157.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T00:30:27.675 INFO:teuthology.orchestra.run.smithi157.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T00:30:27.675 INFO:teuthology.orchestra.run.smithi157.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T00:30:27.677 INFO:teuthology.orchestra.run.smithi157.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T00:30:27.678 INFO:teuthology.orchestra.run.smithi157.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T00:30:30.386 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T00:30:30.521 INFO:teuthology.orchestra.run.smithi157.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T00:30:33.147 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T00:30:33.411 INFO:teuthology.orchestra.run.smithi157.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T00:30:33.714 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T00:30:33.962 INFO:teuthology.orchestra.run.smithi157.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T00:30:34.004 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T00:30:34.033 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T00:30:34.178 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T00:30:34.214 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T00:30:34.297 INFO:teuthology.orchestra.run.smithi157.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T00:30:34.312 INFO:teuthology.orchestra.run.smithi157.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T00:30:34.435 INFO:teuthology.orchestra.run.smithi157.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T00:30:34.486 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T00:30:34.494 INFO:teuthology.orchestra.run.smithi157.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T00:30:34.774 INFO:teuthology.orchestra.run.smithi157.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T00:30:36.982 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T00:30:37.011 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T00:30:37.134 INFO:teuthology.orchestra.run.smithi157.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T00:30:37.138 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T00:30:37.146 INFO:teuthology.orchestra.run.smithi157.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T00:30:37.275 INFO:teuthology.orchestra.run.smithi157.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T00:30:42.146 INFO:teuthology.orchestra.run.smithi157.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T00:30:42.856 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T00:30:42.868 INFO:teuthology.orchestra.run.smithi157.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T00:30:42.900 INFO:teuthology.orchestra.run.smithi157.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T00:30:42.926 INFO:teuthology.orchestra.run.smithi157.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T00:30:43.106 INFO:teuthology.orchestra.run.smithi157.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T00:30:44.034 INFO:teuthology.orchestra.run.smithi157.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T00:30:44.149 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T00:30:44.201 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T00:30:44.204 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T00:30:44.577 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T00:30:46.088 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T00:30:46.879 INFO:teuthology.orchestra.run.smithi157.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T00:30:46.946 INFO:teuthology.orchestra.run.smithi157.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T00:30:46.961 INFO:teuthology.orchestra.run.smithi157.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T00:30:47.104 INFO:teuthology.orchestra.run.smithi157.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T00:30:47.106 INFO:teuthology.orchestra.run.smithi157.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T00:30:47.110 INFO:teuthology.orchestra.run.smithi157.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T00:30:47.250 INFO:teuthology.orchestra.run.smithi157.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T00:30:47.871 INFO:teuthology.orchestra.run.smithi157.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T00:30:47.871 INFO:teuthology.orchestra.run.smithi157.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T00:30:50.643 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T00:30:50.714 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T00:30:50.752 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T00:30:51.097 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T00:30:51.099 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T00:30:51.154 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T00:30:51.393 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T00:30:52.571 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T00:30:52.572 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T00:30:52.720 INFO:teuthology.orchestra.run.smithi157.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T00:30:52.724 INFO:teuthology.orchestra.run.smithi157.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T00:30:52.790 INFO:teuthology.orchestra.run.smithi157.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T00:30:54.262 INFO:teuthology.orchestra.run.smithi157.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T00:30:54.891 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 209 MB in 28s (7,478 kB/s) 2023-09-30T00:30:55.062 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T00:30:55.119 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T00:30:55.123 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T00:30:55.236 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T00:30:55.542 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:30:55.650 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T00:30:56.387 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:30:56.514 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T00:30:57.176 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T00:30:57.197 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:30:57.234 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:30:57.578 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rados. 2023-09-30T00:30:57.597 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:30:57.637 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:30:57.973 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T00:30:57.988 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:30:58.086 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:30:58.674 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T00:30:58.726 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T00:30:58.727 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T00:30:59.007 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T00:30:59.026 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:30:59.067 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:30:59.395 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T00:30:59.411 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:30:59.453 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:30:59.758 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T00:30:59.764 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T00:30:59.782 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T00:30:59.822 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T00:31:00.100 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T00:31:00.118 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:00.167 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:00.392 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 209 MB in 33s (6,252 kB/s) 2023-09-30T00:31:00.436 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T00:31:00.454 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T00:31:00.486 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T00:31:00.568 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T00:31:00.624 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T00:31:00.628 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T00:31:00.742 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T00:31:00.822 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T00:31:00.841 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T00:31:00.897 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T00:31:01.049 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:01.172 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T00:31:01.259 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T00:31:01.275 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T00:31:01.325 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T00:31:01.636 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T00:31:01.652 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T00:31:01.686 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T00:31:01.868 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:01.994 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T00:31:02.039 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T00:31:02.055 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T00:31:02.097 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T00:31:02.543 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T00:31:02.561 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:31:02.603 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:31:02.681 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T00:31:02.698 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:02.739 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:03.042 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2023-09-30T00:31:03.061 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:03.091 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:03.114 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T00:31:03.133 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:31:03.172 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:31:03.428 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T00:31:03.446 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:03.486 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:03.531 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T00:31:03.547 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:31:03.589 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:31:03.772 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T00:31:03.788 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:03.830 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:03.985 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T00:31:04.004 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T00:31:04.034 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T00:31:04.183 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T00:31:04.199 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:04.232 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:04.330 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librgw2. 2023-09-30T00:31:04.349 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:04.388 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:04.502 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T00:31:04.521 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T00:31:04.594 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T00:31:04.796 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T00:31:04.814 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:04.854 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:05.098 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T00:31:05.115 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:05.148 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:05.216 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T00:31:05.232 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T00:31:05.273 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T00:31:05.485 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liboath0. 2023-09-30T00:31:05.502 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T00:31:05.543 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T00:31:05.654 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T00:31:05.671 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T00:31:05.703 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T00:31:05.887 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T00:31:05.903 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:05.946 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:06.006 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T00:31:06.025 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T00:31:06.064 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T00:31:06.416 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-common. 2023-09-30T00:31:06.432 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:06.443 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T00:31:06.459 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T00:31:06.474 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:06.509 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T00:31:06.821 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T00:31:06.838 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T00:31:06.871 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T00:31:07.301 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T00:31:07.321 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:31:07.369 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:31:07.830 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T00:31:07.850 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:31:07.880 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:31:08.231 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T00:31:08.249 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:31:08.289 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:31:08.693 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T00:31:08.708 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T00:31:08.750 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T00:31:09.095 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2023-09-30T00:31:09.096 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-base. 2023-09-30T00:31:09.111 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:09.112 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:09.153 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:09.165 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:09.949 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T00:31:09.967 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:10.007 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:10.151 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T00:31:10.167 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T00:31:10.209 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T00:31:10.377 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0. 2023-09-30T00:31:10.396 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T00:31:10.435 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T00:31:10.613 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T00:31:10.630 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T00:31:10.671 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T00:31:10.696 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T00:31:10.712 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:10.746 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:10.915 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T00:31:10.934 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:10.973 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:11.208 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2023-09-30T00:31:11.226 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:11.266 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:11.355 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T00:31:11.371 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T00:31:11.413 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T00:31:11.758 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T00:31:11.774 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T00:31:11.815 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T00:31:12.136 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T00:31:12.152 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T00:31:12.514 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T00:31:12.533 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T00:31:12.572 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T00:31:12.884 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T00:31:12.899 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T00:31:12.942 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T00:31:13.245 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T00:31:13.260 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T00:31:13.303 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T00:31:13.631 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-mako. 2023-09-30T00:31:13.647 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T00:31:13.689 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T00:31:13.947 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2023-09-30T00:31:13.966 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:14.016 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:14.018 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T00:31:14.037 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T00:31:14.076 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T00:31:14.671 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T00:31:14.687 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T00:31:14.729 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T00:31:15.002 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T00:31:15.019 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T00:31:15.023 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webob. 2023-09-30T00:31:15.039 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T00:31:15.060 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T00:31:15.082 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T00:31:15.446 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T00:31:15.463 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T00:31:15.464 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T00:31:15.482 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T00:31:15.503 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T00:31:15.522 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T00:31:15.789 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T00:31:15.809 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T00:31:15.848 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T00:31:16.092 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T00:31:16.108 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:16.150 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:16.201 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T00:31:16.217 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T00:31:16.262 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T00:31:16.573 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T00:31:16.589 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T00:31:16.629 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T00:31:16.631 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T00:31:16.645 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T00:31:16.687 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T00:31:16.934 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T00:31:16.951 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T00:31:16.989 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-paste. 2023-09-30T00:31:16.992 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T00:31:17.007 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T00:31:17.048 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T00:31:17.346 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T00:31:17.361 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T00:31:17.376 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T00:31:17.394 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T00:31:17.404 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T00:31:17.434 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T00:31:17.716 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T00:31:17.731 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T00:31:17.745 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T00:31:17.763 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T00:31:17.774 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T00:31:17.803 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T00:31:18.086 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T00:31:18.089 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T00:31:18.103 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T00:31:18.109 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T00:31:18.148 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T00:31:18.169 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T00:31:18.472 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T00:31:18.484 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T00:31:18.488 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T00:31:18.500 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T00:31:18.530 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T00:31:18.533 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T00:31:18.825 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-mako. 2023-09-30T00:31:18.842 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T00:31:18.880 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T00:31:18.883 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T00:31:18.897 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T00:31:18.938 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T00:31:19.245 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T00:31:19.263 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T00:31:19.303 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T00:31:19.308 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T00:31:19.324 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:19.366 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:19.590 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T00:31:19.609 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T00:31:19.648 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T00:31:19.702 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T00:31:19.721 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:19.760 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:19.926 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webob. 2023-09-30T00:31:19.946 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T00:31:19.984 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T00:31:20.297 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T00:31:20.315 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:20.337 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T00:31:20.353 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T00:31:20.355 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:20.395 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T00:31:20.690 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T00:31:20.709 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T00:31:20.748 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T00:31:21.127 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T00:31:21.143 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T00:31:21.188 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T00:31:21.530 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T00:31:21.533 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T00:31:21.547 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T00:31:21.553 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:21.591 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:21.597 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T00:31:21.900 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-paste. 2023-09-30T00:31:21.918 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T00:31:21.983 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T00:31:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T00:31:22.311 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T00:31:22.353 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T00:31:22.664 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T00:31:22.680 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T00:31:22.722 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T00:31:23.051 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T00:31:23.067 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T00:31:23.109 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T00:31:23.420 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T00:31:23.438 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T00:31:23.478 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T00:31:23.825 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T00:31:23.841 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T00:31:23.900 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T00:31:24.245 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T00:31:24.261 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:24.303 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:24.344 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph. 2023-09-30T00:31:24.360 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:24.411 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:24.665 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T00:31:24.680 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:24.723 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:24.738 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T00:31:24.755 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:24.797 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:25.160 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T00:31:25.176 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:25.218 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:25.241 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T00:31:25.257 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:25.299 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:25.811 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephadm. 2023-09-30T00:31:25.827 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:25.869 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:26.205 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T00:31:26.221 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T00:31:26.263 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T00:31:26.380 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T00:31:26.396 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:26.438 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:26.566 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T00:31:26.581 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:26.624 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:26.993 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T00:31:27.010 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T00:31:27.051 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T00:31:27.287 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T00:31:27.303 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T00:31:27.345 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T00:31:27.623 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T00:31:27.642 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T00:31:27.681 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T00:31:28.034 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-routes. 2023-09-30T00:31:28.049 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T00:31:28.091 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T00:31:28.420 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T00:31:28.435 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:28.486 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:28.964 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T00:31:28.980 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T00:31:29.022 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T00:31:29.192 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2023-09-30T00:31:29.210 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:29.250 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:29.325 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T00:31:29.342 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T00:31:29.383 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T00:31:29.569 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T00:31:29.585 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:29.652 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:29.703 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T00:31:29.719 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T00:31:29.761 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T00:31:30.056 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T00:31:30.074 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:30.114 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:30.617 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephadm. 2023-09-30T00:31:30.633 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:30.667 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:30.970 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T00:31:30.990 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T00:31:31.028 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T00:31:31.078 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T00:31:31.094 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T00:31:31.136 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T00:31:31.390 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T00:31:31.407 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:31.448 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:31.656 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T00:31:31.674 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T00:31:31.714 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T00:31:31.810 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T00:31:31.829 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T00:31:31.868 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T00:31:32.076 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T00:31:32.094 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T00:31:32.104 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T00:31:32.121 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T00:31:32.134 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T00:31:32.162 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T00:31:32.423 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T00:31:32.440 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T00:31:32.473 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T00:31:32.817 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T00:31:32.818 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-routes. 2023-09-30T00:31:32.835 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T00:31:32.837 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:32.875 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:32.875 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T00:31:33.204 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T00:31:33.220 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:33.262 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:33.766 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T00:31:33.785 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T00:31:33.824 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T00:31:33.986 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T00:31:34.005 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T00:31:34.053 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T00:31:34.145 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T00:31:34.160 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T00:31:34.201 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T00:31:34.355 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T00:31:34.371 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T00:31:34.413 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T00:31:34.538 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T00:31:34.555 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T00:31:34.596 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T00:31:34.775 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T00:31:34.791 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T00:31:34.833 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T00:31:35.144 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T00:31:35.160 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T00:31:35.202 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T00:31:35.505 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T00:31:35.523 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T00:31:35.583 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T00:31:35.847 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T00:31:35.866 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T00:31:35.905 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T00:31:35.908 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T00:31:35.928 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T00:31:35.967 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T00:31:36.269 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T00:31:36.289 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T00:31:36.327 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T00:31:36.451 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T00:31:36.470 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T00:31:36.509 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T00:31:36.681 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T00:31:36.697 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T00:31:36.739 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T00:31:36.887 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T00:31:36.904 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T00:31:36.945 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T00:31:37.033 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T00:31:37.053 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T00:31:37.091 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T00:31:37.402 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T00:31:37.419 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T00:31:37.460 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T00:31:37.629 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T00:31:37.650 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:37.687 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:37.998 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T00:31:38.016 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:38.056 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:38.478 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T00:31:38.498 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T00:31:38.528 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T00:31:38.807 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T00:31:38.827 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T00:31:38.865 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T00:31:38.898 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T00:31:38.915 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T00:31:38.956 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:31:39.168 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T00:31:39.186 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T00:31:39.225 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package jq. 2023-09-30T00:31:39.243 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T00:31:39.251 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T00:31:39.283 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:31:39.553 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package socat. 2023-09-30T00:31:39.569 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T00:31:39.579 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T00:31:39.596 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T00:31:39.610 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T00:31:39.638 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T00:31:39.923 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T00:31:39.939 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T00:31:39.949 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T00:31:39.965 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T00:31:39.981 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T00:31:40.007 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T00:31:40.319 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-test. 2023-09-30T00:31:40.336 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:40.352 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T00:31:40.368 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T00:31:40.402 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:40.410 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T00:31:40.730 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T00:31:40.748 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T00:31:40.788 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T00:31:41.083 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T00:31:41.102 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T00:31:41.158 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T00:31:41.462 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T00:31:41.482 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T00:31:41.519 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T00:31:41.806 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T00:31:41.826 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T00:31:41.864 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T00:31:42.201 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T00:31:42.218 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T00:31:42.259 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T00:31:42.757 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T00:31:42.774 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:42.813 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:43.269 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T00:31:43.285 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T00:31:43.327 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T00:31:43.681 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T00:31:43.697 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T00:31:43.738 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:31:44.000 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2023-09-30T00:31:44.017 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T00:31:44.083 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:31:44.336 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2023-09-30T00:31:44.354 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T00:31:44.394 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T00:31:44.715 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T00:31:44.731 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T00:31:44.773 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T00:31:45.172 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2023-09-30T00:31:45.188 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:45.230 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:45.514 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T00:31:45.531 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:45.572 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:45.893 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T00:31:45.911 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T00:31:45.951 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:31:46.450 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T00:31:46.466 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:46.505 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:46.891 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T00:31:46.908 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T00:31:46.967 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T00:31:47.318 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T00:31:47.338 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T00:31:47.376 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:31:48.956 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T00:31:48.977 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T00:31:49.014 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T00:31:49.359 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T00:31:49.376 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T00:31:49.417 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T00:31:49.721 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-attr. 2023-09-30T00:31:49.738 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T00:31:49.779 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T00:31:50.107 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T00:31:50.127 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T00:31:50.151 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T00:31:50.164 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T00:31:50.167 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:31:50.209 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:50.467 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T00:31:50.485 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T00:31:50.522 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T00:31:50.540 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T00:31:50.580 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:31:50.609 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T00:31:50.995 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T00:31:51.013 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T00:31:51.053 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T00:31:51.101 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T00:31:51.119 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:51.159 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:51.498 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T00:31:51.518 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T00:31:51.537 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T00:31:51.555 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T00:31:51.556 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T00:31:51.613 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T00:31:51.859 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T00:31:51.876 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T00:31:51.934 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T00:31:51.967 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T00:31:51.986 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T00:31:52.025 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:31:52.279 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T00:31:52.296 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T00:31:52.336 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T00:31:52.648 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T00:31:52.665 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T00:31:52.706 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T00:31:53.067 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-py. 2023-09-30T00:31:53.084 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T00:31:53.125 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T00:31:53.453 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T00:31:53.472 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T00:31:53.511 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T00:31:53.711 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T00:31:53.729 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T00:31:53.794 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T00:31:53.797 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T00:31:53.818 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T00:31:53.880 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T00:31:54.114 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T00:31:54.135 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T00:31:54.172 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T00:31:54.183 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T00:31:54.201 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T00:31:54.250 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T00:31:54.485 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-attr. 2023-09-30T00:31:54.502 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T00:31:54.543 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T00:31:54.561 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T00:31:54.582 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T00:31:54.619 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T00:31:54.913 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T00:31:54.922 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T00:31:54.932 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T00:31:54.942 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T00:31:54.971 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T00:31:54.980 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T00:31:55.274 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T00:31:55.291 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T00:31:55.332 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T00:31:55.669 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T00:31:55.688 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T00:31:55.709 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package radosgw. 2023-09-30T00:31:55.726 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:55.752 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T00:31:55.800 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:56.214 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T00:31:56.234 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T00:31:56.272 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T00:31:56.558 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T00:31:56.578 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T00:31:56.616 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T00:31:56.970 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T00:31:56.990 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T00:31:57.028 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T00:31:57.331 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T00:31:57.348 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T00:31:57.389 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T00:31:57.546 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T00:31:57.564 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:31:57.604 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:57.742 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-py. 2023-09-30T00:31:57.760 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T00:31:57.800 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T00:31:57.930 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T00:31:58.129 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T00:31:58.145 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T00:31:58.195 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T00:31:58.225 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T00:31:58.351 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T00:31:58.477 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T00:31:58.490 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T00:31:58.510 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T00:31:58.548 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T00:31:58.696 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:31:58.884 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T00:31:58.901 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T00:31:58.942 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T00:31:59.288 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T00:31:59.305 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T00:31:59.333 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user cephadm....done 2023-09-30T00:31:59.346 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T00:31:59.494 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:31:59.638 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T00:31:59.657 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T00:31:59.675 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T00:31:59.715 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T00:31:59.992 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T00:32:00.276 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:32:00.402 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T00:32:00.444 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2023-09-30T00:32:00.461 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:32:00.520 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:00.528 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T00:32:00.863 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T00:32:01.281 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T00:32:01.600 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T00:32:01.884 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T00:32:02.168 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T00:32:02.241 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T00:32:02.257 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:32:02.298 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:02.469 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:02.624 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T00:32:02.788 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T00:32:02.904 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T00:32:02.913 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T00:32:03.038 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T00:32:03.164 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T00:32:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T00:32:03.358 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:03.499 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T00:32:03.625 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T00:32:03.953 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user cephadm....done 2023-09-30T00:32:04.009 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T00:32:04.090 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T00:32:04.330 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T00:32:04.563 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T00:32:04.614 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T00:32:04.848 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:32:04.924 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T00:32:04.974 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T00:32:05.100 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T00:32:05.209 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T00:32:05.418 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T00:32:05.544 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T00:32:05.820 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T00:32:05.828 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T00:32:06.105 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T00:32:06.145 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T00:32:06.271 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T00:32:06.398 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T00:32:06.416 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T00:32:06.674 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T00:32:06.717 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T00:32:06.967 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:07.135 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T00:32:07.276 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T00:32:07.395 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T00:32:07.520 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T00:32:07.695 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T00:32:07.871 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T00:32:07.980 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T00:32:07.997 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:08.106 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T00:32:08.324 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:32:08.458 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T00:32:08.507 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T00:32:08.800 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T00:32:08.801 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T00:32:08.927 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T00:32:09.053 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T00:32:09.072 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T00:32:09.346 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T00:32:09.425 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T00:32:09.647 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:32:09.708 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T00:32:09.773 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T00:32:09.899 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T00:32:10.035 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T00:32:10.242 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T00:32:10.344 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T00:32:10.535 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T00:32:10.630 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T00:32:10.766 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T00:32:10.827 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T00:32:10.892 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T00:32:10.953 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T00:32:11.186 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T00:32:11.246 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T00:32:11.514 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T00:32:11.571 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T00:32:11.815 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T00:32:11.931 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T00:32:12.248 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T00:32:12.308 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T00:32:12.434 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:12.509 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T00:32:12.637 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T00:32:12.743 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:32:12.886 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T00:32:12.964 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T00:32:13.221 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T00:32:13.324 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T00:32:13.346 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T00:32:13.449 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T00:32:13.472 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T00:32:13.567 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T00:32:13.757 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T00:32:13.893 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T00:32:14.075 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:32:14.186 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:14.201 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T00:32:14.315 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T00:32:14.326 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T00:32:14.653 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T00:32:14.954 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T00:32:15.255 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T00:32:15.381 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T00:32:15.682 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T00:32:15.770 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:15.896 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:32:15.983 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T00:32:16.056 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T00:32:16.293 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T00:32:16.634 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T00:32:16.689 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T00:32:16.921 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T00:32:16.969 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T00:32:17.046 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T00:32:17.261 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:32:17.375 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T00:32:17.387 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T00:32:17.735 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T00:32:17.738 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:17.869 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T00:32:17.884 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T00:32:17.995 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T00:32:18.216 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:18.313 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T00:32:18.334 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T00:32:18.606 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:18.685 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T00:32:18.731 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T00:32:19.740 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:19.891 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T00:32:20.009 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T00:32:20.126 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:32:20.252 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T00:32:20.257 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:20.383 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:32:20.509 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T00:32:20.545 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T00:32:20.671 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T00:32:20.879 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T00:32:21.005 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:21.131 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:21.172 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T00:32:21.355 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T00:32:21.355 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T00:32:21.456 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:32:21.582 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T00:32:21.824 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:21.967 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:22.044 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T00:32:22.092 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T00:32:22.329 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T00:32:22.411 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:22.537 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T00:32:22.655 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T00:32:23.024 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:23.046 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T00:32:23.158 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:23.268 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:23.394 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:23.520 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T00:32:23.821 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T00:32:24.114 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:24.126 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:24.240 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:24.252 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T00:32:24.366 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T00:32:24.378 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T00:32:24.496 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:32:24.647 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T00:32:24.940 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T00:32:25.066 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T00:32:25.269 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:25.367 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:25.513 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:25.745 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T00:32:25.745 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T00:32:25.813 INFO:teuthology.orchestra.run.smithi157.stdout:Adding group ceph....done 2023-09-30T00:32:26.190 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user ceph....done 2023-09-30T00:32:26.194 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:26.340 INFO:teuthology.orchestra.run.smithi157.stdout:Setting system user ceph properties....done 2023-09-30T00:32:26.360 INFO:teuthology.orchestra.run.smithi157.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T00:32:26.405 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T00:32:26.511 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T00:32:26.723 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T00:32:26.838 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T00:32:27.041 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T00:32:27.304 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:27.377 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:27.429 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:27.528 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:27.654 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:27.725 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T00:32:27.725 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T00:32:27.780 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:27.906 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T00:32:28.190 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T00:32:28.389 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:28.492 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:28.593 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:28.694 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T00:32:28.797 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T00:32:29.268 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:29.483 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T00:32:29.485 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T00:32:29.605 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:29.937 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:30.125 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2023-09-30T00:32:30.160 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T00:32:30.160 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T00:32:30.535 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2023-09-30T00:32:30.638 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:30.711 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2023-09-30T00:32:30.736 INFO:teuthology.orchestra.run.smithi110.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T00:32:30.885 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T00:32:31.120 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T00:32:31.120 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T00:32:31.198 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T00:32:31.581 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:31.674 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:31.708 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:32.080 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T00:32:32.080 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T00:32:32.117 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:32.463 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T00:32:32.463 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T00:32:32.718 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:32.911 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:33.054 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:33.098 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T00:32:33.188 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:33.356 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:33.486 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T00:32:33.553 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:33.621 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T00:32:33.769 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T00:32:33.769 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T00:32:33.905 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T00:32:34.196 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:34.426 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T00:32:34.427 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T00:32:34.864 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:35.357 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T00:32:35.357 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T00:32:35.783 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:35.909 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:36.093 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:36.322 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T00:32:36.322 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T00:32:36.758 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T00:32:36.769 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:36.904 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:37.063 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:37.206 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:32:37.333 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T00:32:37.411 INFO:teuthology.orchestra.run.smithi157.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T00:32:37.414 DEBUG:teuthology.parallel:result is None 2023-09-30T00:32:37.493 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T00:32:37.780 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T00:32:40.558 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T00:32:41.135 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T00:32:41.137 DEBUG:teuthology.parallel:result is None 2023-09-30T00:32:41.138 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:32:41.279 DEBUG:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2023-09-30T00:32:41.302 INFO:teuthology.orchestra.run.smithi110.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:32:41.302 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:32:41.303 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T00:32:41.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:32:41.516 DEBUG:teuthology.orchestra.run.smithi157:> dpkg-query -W -f '${Version}' ceph 2023-09-30T00:32:41.541 INFO:teuthology.orchestra.run.smithi157.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:32:41.541 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:32:41.541 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T00:32:41.541 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:32:41.542 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:41.542 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:32:41.558 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:41.559 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:32:41.599 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:32:41.599 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:41.599 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:32:41.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:32:41.672 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:41.672 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:32:41.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:32:41.703 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:32:41.703 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:41.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:32:41.719 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:32:41.730 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:41.730 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:32:41.746 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:32:41.802 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:32:41.803 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:41.803 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:32:41.819 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:32:41.828 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:41.828 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:32:41.842 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:32:41.854 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:32:41.916 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:32:41.917 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2023-09-30T00:32:41.919 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2023-09-30T00:32:41.935 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:32:41.935 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:32:41.978 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:32:41.993 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:32:41.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:32:41.996 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:32:42.020 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:32:42.020 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:32:42.020 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406725', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'must scrub before tier agent can activate'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache readproxy', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 3600', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:32:42.021 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:32:42.050 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:32:42.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:32:42.060 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:32:42.073 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:42.073 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:32:42.118 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:32:42.119 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 00:28:44.959323925 +0000 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 00:28:44.695330011 +0000 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 00:28:44.695330011 +0000 2023-09-30T00:32:42.125 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T00:32:42.125 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:32:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T00:32:42.140 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T00:32:42.141 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000268789 s, 1.9 MB/s 2023-09-30T00:32:42.141 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:32:42.150 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2023-09-30T00:32:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:32:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.157 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 366 Links: 1 2023-09-30T00:32:42.157 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.158 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 00:28:45.439312863 +0000 2023-09-30T00:32:42.158 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 00:28:45.183318763 +0000 2023-09-30T00:32:42.158 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 00:28:45.183318763 +0000 2023-09-30T00:32:42.158 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T00:32:42.158 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:32:42.213 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T00:32:42.213 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T00:32:42.213 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000260035 s, 2.0 MB/s 2023-09-30T00:32:42.214 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:32:42.223 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 00:28:45.967300691 +0000 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 00:28:45.675307422 +0000 2023-09-30T00:32:42.230 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 00:28:45.675307422 +0000 2023-09-30T00:32:42.231 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T00:32:42.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:32:42.242 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T00:32:42.242 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T00:32:42.243 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.0002617 s, 2.0 MB/s 2023-09-30T00:32:42.244 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:32:42.292 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2023-09-30T00:32:42.300 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:32:42.300 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.300 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 379 Links: 1 2023-09-30T00:32:42.300 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.300 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 00:28:46.439289808 +0000 2023-09-30T00:32:42.301 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 00:28:46.167296081 +0000 2023-09-30T00:32:42.301 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 00:28:46.167296081 +0000 2023-09-30T00:32:42.301 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T00:32:42.301 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:32:42.316 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T00:32:42.316 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T00:32:42.316 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000291963 s, 1.8 MB/s 2023-09-30T00:32:42.317 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:32:42.368 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:42.369 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:42.369 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:32:42.375 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:32:42.376 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2023-09-30T00:32:42.426 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:32:42.426 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.427 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 359 Links: 1 2023-09-30T00:32:42.427 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.427 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 00:28:44.960796955 +0000 2023-09-30T00:32:42.427 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:28:44.652804046 +0000 2023-09-30T00:32:42.427 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:28:44.652804046 +0000 2023-09-30T00:32:42.427 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T00:32:42.427 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:32:42.438 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T00:32:42.438 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T00:32:42.438 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000300596 s, 1.7 MB/s 2023-09-30T00:32:42.439 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:32:42.448 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2023-09-30T00:32:42.498 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 00:28:45.476785079 +0000 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:28:45.188791707 +0000 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:28:45.188791707 +0000 2023-09-30T00:32:42.499 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T00:32:42.499 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:32:42.510 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T00:32:42.510 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T00:32:42.510 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000296011 s, 1.7 MB/s 2023-09-30T00:32:42.511 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:32:42.519 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2023-09-30T00:32:42.566 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:32:42.566 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.567 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-09-30T00:32:42.567 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.567 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 00:28:45.984773385 +0000 2023-09-30T00:32:42.567 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:28:45.696780015 +0000 2023-09-30T00:32:42.567 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:28:45.696780015 +0000 2023-09-30T00:32:42.567 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T00:32:42.567 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:32:42.578 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T00:32:42.578 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T00:32:42.578 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000292444 s, 1.8 MB/s 2023-09-30T00:32:42.579 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:32:42.587 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2023-09-30T00:32:42.638 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:32:42.638 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:32:42.639 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-09-30T00:32:42.639 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:32:42.639 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-09-30 00:28:46.480761968 +0000 2023-09-30T00:32:42.639 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-09-30 00:28:46.212768137 +0000 2023-09-30T00:32:42.639 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-09-30 00:28:46.212768137 +0000 2023-09-30T00:32:42.639 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-09-30T00:32:42.639 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:32:42.653 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-09-30T00:32:42.653 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-09-30T00:32:42.653 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000297697 s, 1.7 MB/s 2023-09-30T00:32:42.654 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:32:42.662 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:42.662 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:32:42.662 INFO:tasks.ceph:Generating config... 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd backoff on degraded = True 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd backoff on peering = True 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd map cache size = 1 2023-09-30T00:32:42.664 INFO:tasks.ceph:[osd] osd max backfills = 6 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd max scrubs = 2 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd objectstore = filestore 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd sloppy crc = True 2023-09-30T00:32:42.665 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2023-09-30T00:32:42.666 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] ms type = async 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-09-30T00:32:42.666 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2023-09-30T00:32:42.666 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:32:42.666 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:32:42.666 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:32:42.666 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T00:32:42.667 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T00:32:42.667 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:32:42.667 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:32:42.702 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:32:42.704 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:32:42.738 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:32:42.795 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.c', '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]'), ('mon.b', '172.21.15.157')] 2023-09-30T00:32:42.796 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.157', 'mon election default strategy': 1, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T00:32:42.797 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:42.797 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:32:42.802 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.110 --addv c '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]' --add b 172.21.15.157 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:42.840 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:42.840 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid df8ee23b-92e0-405f-810c-7a3d9ae641bd 2023-09-30T00:32:42.840 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = octopus 2023-09-30T00:32:42.840 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:fsid df8ee23b-92e0-405f-810c-7a3d9ae641bd 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2023-09-30T00:32:42.838545+0000 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:created 2023-09-30T00:32:42.838545+0000 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 15 (octopus) 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2023-09-30T00:32:42.841 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T00:32:42.843 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:32:42.890 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID df8ee23b-92e0-405f-810c-7a3d9ae641bd... 2023-09-30T00:32:42.892 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:32:42.894 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2023-09-30T00:32:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:32:42.929 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:32:42.930 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.931 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.157" 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug inject dispatch delay duration = 0.1 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug inject dispatch delay probability = 0.1 2023-09-30T00:32:42.932 INFO:teuthology.orchestra.run.smithi110.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_min_size = 2 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_size = 2 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = df8ee23b-92e0-405f-810c-7a3d9ae641bd 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 120 2023-09-30T00:32:42.933 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2023-09-30T00:32:42.934 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2023-09-30T00:32:42.935 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd backoff on degraded = True 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd backoff on peering = True 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd inject bad map crc probability = 0.1 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd map cache size = 1 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd max backfills = 6 2023-09-30T00:32:42.936 INFO:teuthology.orchestra.run.smithi110.stdout: osd max markdown count = 1000 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd max scrubs = 2 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = filestore 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub during recovery = False 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub min interval = 60 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd sloppy crc = True 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd_max_pg_log_entries = 3000 2023-09-30T00:32:42.937 INFO:teuthology.orchestra.run.smithi110.stdout: osd_min_pg_log_entries = 3000 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2023-09-30T00:32:42.938 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2023-09-30T00:32:42.941 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:32:42.942 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:42.943 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2023-09-30T00:32:42.944 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2023-09-30T00:32:42.946 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2023-09-30T00:32:42.947 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2023-09-30T00:32:42.948 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.157" 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: mon election default strategy = 1 2023-09-30T00:32:42.949 INFO:teuthology.orchestra.run.smithi157.stdout: ms type = async 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug inject dispatch delay duration = 0.1 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug inject dispatch delay probability = 0.1 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_min_size = 2 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_size = 2 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = df8ee23b-92e0-405f-810c-7a3d9ae641bd 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.950 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 120 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.951 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2023-09-30T00:32:42.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore zero block detection = True 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd backoff on degraded = True 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd backoff on peering = True 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify cached snaps = True 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify missing on start = True 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd inject bad map crc probability = 0.1 2023-09-30T00:32:42.953 INFO:teuthology.orchestra.run.smithi157.stdout: osd map cache size = 1 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd max backfills = 6 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd max scrubs = 2 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock override recovery settings = True 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = filestore 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue = debug_random 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue cut off = debug_random 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub during recovery = False 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub min interval = 60 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd sloppy crc = True 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_pg_log_entries = 3000 2023-09-30T00:32:42.954 INFO:teuthology.orchestra.run.smithi157.stdout: osd_min_pg_log_entries = 3000 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2023-09-30T00:32:42.955 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:32:42.956 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon min osdmap epochs = 50 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune interval = 2 2023-09-30T00:32:42.957 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune min = 15 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: mon scrub interval = 300 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: paxos service trim min = 10 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2023-09-30T00:32:42.958 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2023-09-30T00:32:42.959 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2023-09-30T00:32:42.959 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:32:42.959 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:32:43.039 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:32:43.040 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:43.040 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:32:43.047 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:43.047 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:32:43.053 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:32:43.053 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:43.054 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:32:43.054 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:32:43.078 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:43.078 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:43.126 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:32:43.127 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:43.127 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:32:43.127 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:32:43.153 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:43.153 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:43.160 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:32:43.160 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:32:43.160 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:32:43.205 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:32:43.207 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:32:43.252 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:32:43.255 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:32:43.255 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:32:43.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:32:43.292 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:32:43.305 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T00:32:43.339 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T00:32:43.352 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:32:43.353 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:32:43.353 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:32:43.367 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:43.367 INFO:tasks.ceph:role: osd.0 2023-09-30T00:32:43.368 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:32:43.368 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:32:43.411 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-09-30T00:32:43.411 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:43.412 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:43.412 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:43.412 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:43.412 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:43.412 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:43.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T00:32:43.413 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:32:43.432 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:32:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:43.487 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:43.488 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:32:43.498 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:43.499 INFO:tasks.ceph:role: osd.1 2023-09-30T00:32:43.499 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:32:43.499 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:43.539 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:43.540 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:43.540 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:43.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T00:32:43.541 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T00:32:43.605 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:32:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:43.615 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:43.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:32:43.626 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:43.626 INFO:tasks.ceph:role: osd.2 2023-09-30T00:32:43.626 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:32:43.627 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:43.713 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:43.714 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:43.714 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:43.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T00:32:43.715 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T00:32:43.731 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:32:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:43.740 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:43.740 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:32:43.750 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:43.750 INFO:tasks.ceph:role: osd.3 2023-09-30T00:32:43.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:32:43.750 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:32:43.832 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:43.832 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:43.832 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:43.833 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:43.834 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T00:32:43.834 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T00:32:43.855 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:32:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:43.866 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:43.866 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:43.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:43.979+0000 7f2e1c81f3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:32:43.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:43.979+0000 7f2e1c81f3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:32:43.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:43.991+0000 7f2e1c81f3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:43.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:43.995+0000 7f2e1c81f3c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:44.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.067+0000 7f2e1c81f3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.067+0000 7f2e1c81f3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.067+0000 7f2e1c81f3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.087 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:32:44.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:44.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.135+0000 7f395e8cd3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:32:44.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.135+0000 7f395e8cd3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:32:44.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.147+0000 7f395e8cd3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.147+0000 7f395e8cd3c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.219+0000 7f395e8cd3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.221 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.219+0000 7f395e8cd3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.219+0000 7f395e8cd3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.236 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:32:44.254 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:44.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.291+0000 7ff3810463c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:32:44.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.291+0000 7ff3810463c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:32:44.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.303+0000 7ff3810463c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.303+0000 7ff3810463c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.375+0000 7ff3810463c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.375+0000 7ff3810463c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.375+0000 7ff3810463c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.392 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:32:44.453 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.487+0000 7efd321ca3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:32:44.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.487+0000 7efd321ca3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:32:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.499+0000 7efd321ca3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.499+0000 7efd321ca3c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:44.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.575+0000 7efd321ca3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:44.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.575+0000 7efd321ca3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T00:32:44.575+0000 7efd321ca3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:44.592 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:32:44.606 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T00:32:44.623 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:44.623 INFO:tasks.ceph:role: osd.4 2023-09-30T00:32:44.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:32:44.623 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:32:44.665 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:44.666 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:44.667 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T00:32:44.667 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T00:32:44.689 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T00:32:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:44.743 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:44.744 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T00:32:44.758 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:44.758 INFO:tasks.ceph:role: osd.5 2023-09-30T00:32:44.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:32:44.758 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:32:44.799 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:44.800 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:44.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T00:32:44.801 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T00:32:44.865 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T00:32:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:44.878 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:44.878 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T00:32:44.890 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:44.890 INFO:tasks.ceph:role: osd.6 2023-09-30T00:32:44.890 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:32:44.890 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:32:44.931 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:44.931 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:44.931 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:44.932 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:44.933 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T00:32:44.933 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T00:32:44.950 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T00:32:45.003 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:45.005 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T00:32:45.020 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:32:45.020 INFO:tasks.ceph:role: osd.7 2023-09-30T00:32:45.020 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:32:45.020 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:32:45.063 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:32:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:32:45.064 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:32:45.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2023-09-30T00:32:45.065 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T00:32:45.130 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T00:32:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:32:45.140 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T00:32:45.141 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:45.210 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.206+0000 7f197cc213c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T00:32:45.210 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.206+0000 7f197cc213c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T00:32:45.222 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.218+0000 7f197cc213c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.222 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.218+0000 7f197cc213c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:45.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.290+0000 7f197cc213c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.290+0000 7f197cc213c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.296 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.290+0000 7f197cc213c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.310 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T00:32:45.324 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:45.360 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.354+0000 7f404e5323c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T00:32:45.360 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.358+0000 7f404e5323c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T00:32:45.372 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.370+0000 7f404e5323c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.372 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.370+0000 7f404e5323c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:45.446 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.442+0000 7f404e5323c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.446 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.442+0000 7f404e5323c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.446 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.442+0000 7f404e5323c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.460 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T00:32:45.472 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:45.507 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.502+0000 7fb8aa2203c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T00:32:45.507 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.502+0000 7fb8aa2203c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T00:32:45.519 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.514+0000 7fb8aa2203c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.519 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.514+0000 7fb8aa2203c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:45.592 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.586+0000 7fb8aa2203c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.592 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.586+0000 7fb8aa2203c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.592 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.586+0000 7fb8aa2203c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T00:32:45.618 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:45.705 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.702+0000 7fc00fe9d3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T00:32:45.705 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.702+0000 7fc00fe9d3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T00:32:45.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.714+0000 7fc00fe9d3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.717 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.714+0000 7fc00fe9d3c0 -1 journal FileJournal::_open_file : zeroing journal 2023-09-30T00:32:45.790 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.786+0000 7fc00fe9d3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:45.790 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.786+0000 7fc00fe9d3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.790 INFO:teuthology.orchestra.run.smithi157.stderr:2023-09-30T00:32:45.786+0000 7fc00fe9d3c0 -1 journal do_read_entry(4096): bad header magic 2023-09-30T00:32:45.804 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T00:32:45.817 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:32:45.817 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:45.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T00:32:45.833 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:45.833 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:32:45.847 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:45.848 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:32:45.859 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:45.859 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:32:45.869 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:45.869 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:32:45.884 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:45.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:32:45.900 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:45.901 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T00:32:45.915 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:45.916 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T00:32:45.975 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:45.976 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T00:32:45.991 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:45.991 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T00:32:46.000 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:46.001 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:32:46.007 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:46.007 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T00:32:46.050 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:32:46.050 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:32:46.052 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:32:46.063 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2023-09-30T00:32:46.063 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrbBdl2HcnDBAA298G0q1nOBantOvYBPwNGQ== 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrbBdloxd7OhAAz72wefyIxa37qu8eYfXUwA== 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCsbBdlj94XCBAAZF+w7jtiUtDg5gRY0ejmEQ== 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCsbBdlFClsERAAhREq5rQzssK00qwBqKyJeg== 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCsbBdlU/grHRAAzHW8RaVUdmjEBjD0nf63iA== 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2023-09-30T00:32:46.064 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrbBdl8O/9DhAAPxrABCyY0b+jCXrfZSYYDA== 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCtbBdluzBxDBAAm1mPJZU7EXUyoaXQW8JYPQ== 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCtbBdlYehjFRAAiDZKPxco/xDAOj+YdfpVqA== 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCtbBdlqt8pHhAAGKhMMntb1cXLFNCv7bT5Kw== 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCtbBdl7UnvKRAAdSWq8lXxQTkxOZP0NLyzwA== 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrbBdlrE5bERAAZqzRYChF/GJphQ/KGCzsPA== 2023-09-30T00:32:46.065 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2023-09-30T00:32:46.066 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrbBdlFr0kFBAA47mWtC3HnZz32xUc05h9ng== 2023-09-30T00:32:46.067 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2023-09-30T00:32:46.067 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCrbBdl2HcnDBAA298G0q1nOBantOvYBPwNGQ== 2023-09-30T00:32:46.067 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2023-09-30T00:32:46.067 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCrbBdloxd7OhAAz72wefyIxa37qu8eYfXUwA== 2023-09-30T00:32:46.067 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2023-09-30T00:32:46.067 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCsbBdlj94XCBAAZF+w7jtiUtDg5gRY0ejmEQ== 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCsbBdlFClsERAAhREq5rQzssK00qwBqKyJeg== 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCsbBdlU/grHRAAzHW8RaVUdmjEBjD0nf63iA== 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCrbBdl8O/9DhAAPxrABCyY0b+jCXrfZSYYDA== 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCtbBdluzBxDBAAm1mPJZU7EXUyoaXQW8JYPQ== 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2023-09-30T00:32:46.068 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCtbBdlYehjFRAAiDZKPxco/xDAOj+YdfpVqA== 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCtbBdlqt8pHhAAGKhMMntb1cXLFNCv7bT5Kw== 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCtbBdl7UnvKRAAdSWq8lXxQTkxOZP0NLyzwA== 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCrbBdlrE5bERAAZqzRYChF/GJphQ/KGCzsPA== 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2023-09-30T00:32:46.069 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQCrbBdlFr0kFBAA47mWtC3HnZz32xUc05h9ng== 2023-09-30T00:32:46.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:32:46.072 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:32:46.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.154 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.202 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.204 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.249 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.252 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.293 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.334 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.379 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:32:46.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:32:46.422 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.470 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.517 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.519 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.563 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.565 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.607 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.650 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:32:46.698 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:32:46.700 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:32:46.745 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:32:46.747 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:32:46.789 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:32:46.789 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:32:46.849 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:32:46.904 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:32:46.917 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T00:32:46.932 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:32:46.988 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T00:32:47.002 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T00:32:47.019 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:32:47.075 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T00:32:47.088 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:47.090 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:32:47.095 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:32:47.095 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:32:47.096 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T00:32:47.098 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:32:47.098 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T00:32:47.098 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T00:32:47.100 INFO:tasks.ceph.mon.c:Started 2023-09-30T00:32:47.100 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T00:32:47.100 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T00:32:47.142 INFO:tasks.ceph.mon.b:Started 2023-09-30T00:32:47.142 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:32:47.142 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T00:32:47.142 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T00:32:47.144 INFO:tasks.ceph.mgr.y:Started 2023-09-30T00:32:47.145 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:32:47.145 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T00:32:47.147 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:32:47.147 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:32:47.147 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:32:52.662 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:52.658+0000 7fc9a9750000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:32:52.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:52.674+0000 7f021db34000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:32:53.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.282+0000 7f021db34000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:32:53.289 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.286+0000 7fc9a9750000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:32:53.334 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2023-09-30T00:32:53.346 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.342+0000 7fc9a9750000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:32:53.348 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:32:53.349 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:53.349 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:32:53.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.346+0000 7f021db34000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:32:53.361 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:53.361 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:32:53.372 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:53.372 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:32:53.381 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T00:32:53.381 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:32:53.439 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:53.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T00:32:53.455 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:53.455 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T00:32:53.458 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.454+0000 7fc9a9750000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:32:53.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.458+0000 7f021db34000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:32:53.466 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:53.466 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T00:32:53.521 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-09-30T00:32:53.521 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T00:32:53.534 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 2a37e1e5-0043-46ce-9833-37acd6fd9caa 0 2023-09-30T00:32:53.683 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.678+0000 7fc9a9750000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:32:53.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.686+0000 7f021db34000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:32:53.729 INFO:teuthology.orchestra.run.smithi157.stdout:0 2023-09-30T00:32:53.731 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.726+0000 7fc9a9750000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:32:53.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.734+0000 7f021db34000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:32:53.743 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 71133118-837b-477d-8c8a-f0b6703508e8 1 2023-09-30T00:32:53.907 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.902+0000 7fc9a9750000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:32:53.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.914+0000 7f021db34000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:32:53.936 INFO:teuthology.orchestra.run.smithi157.stdout:1 2023-09-30T00:32:53.947 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 518bc2ab-0374-44c0-95e2-da0383144b93 2 2023-09-30T00:32:53.968 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:53.962+0000 7fc9a9750000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:32:53.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:53.978+0000 7f021db34000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:32:54.031 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:54.026+0000 7fc9a9750000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:32:54.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:54.042+0000 7f021db34000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:32:54.137 INFO:teuthology.orchestra.run.smithi157.stdout:2 2023-09-30T00:32:54.151 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new e22cf827-1bef-4184-8575-54d91d8435c1 3 2023-09-30T00:32:54.391 INFO:teuthology.orchestra.run.smithi157.stdout:3 2023-09-30T00:32:54.403 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 0c705242-5495-4641-9a0c-011a1766dc49 4 2023-09-30T00:32:54.436 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:54.434+0000 7fc9a9750000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:32:54.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:54.490+0000 7f021db34000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:32:54.551 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:54.546+0000 7fc9a9750000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:32:54.591 INFO:teuthology.orchestra.run.smithi157.stdout:4 2023-09-30T00:32:54.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 9f973d43-09c3-4fed-ad02-8f208c64d9f3 5 2023-09-30T00:32:54.615 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:54.610+0000 7f021db34000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:32:54.799 INFO:teuthology.orchestra.run.smithi157.stdout:5 2023-09-30T00:32:54.813 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new b9296c08-c456-43b4-882a-28b4979ea6ee 6 2023-09-30T00:32:54.850 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:54.846+0000 7fc9a9750000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:32:54.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:54.926+0000 7f021db34000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:32:54.997 INFO:teuthology.orchestra.run.smithi157.stdout:6 2023-09-30T00:32:55.009 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 4e612da4-22aa-46e1-8324-d9982302791b 7 2023-09-30T00:32:55.018 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.014+0000 7fc9a9750000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:32:55.085 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.082+0000 7fc9a9750000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:32:55.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.090+0000 7f021db34000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:32:55.136 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.134+0000 7fc9a9750000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:32:55.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.158+0000 7f021db34000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:32:55.194 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.190+0000 7fc9a9750000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:32:55.211 INFO:teuthology.orchestra.run.smithi157.stdout:7 2023-09-30T00:32:55.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.206+0000 7f021db34000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:32:55.224 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:32:55.224 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T00:32:55.226 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:32:55.226 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:32:55.226 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T00:32:55.228 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:32:55.228 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:32:55.229 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T00:32:55.231 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:32:55.231 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:32:55.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T00:32:55.233 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:32:55.233 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T00:32:55.233 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T00:32:55.235 INFO:tasks.ceph.osd.4:Started 2023-09-30T00:32:55.235 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T00:32:55.236 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T00:32:55.237 INFO:tasks.ceph.osd.5:Started 2023-09-30T00:32:55.237 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T00:32:55.237 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-30T00:32:55.239 INFO:tasks.ceph.osd.6:Started 2023-09-30T00:32:55.239 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T00:32:55.239 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-30T00:32:55.241 INFO:tasks.ceph.osd.7:Started 2023-09-30T00:32:55.241 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:32:55.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.278+0000 7f021db34000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:32:55.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:55.286+0000 7f2b6e1e93c0 -1 Falling back to public interface 2023-09-30T00:32:55.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:55.294+0000 7fe96d1e63c0 -1 Falling back to public interface 2023-09-30T00:32:55.299 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:55.294+0000 7f8ffb3ed3c0 -1 Falling back to public interface 2023-09-30T00:32:55.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:55.302+0000 7f5b3a4ac3c0 -1 Falling back to public interface 2023-09-30T00:32:55.312 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:55.310+0000 7fbacb63b3c0 -1 Falling back to public interface 2023-09-30T00:32:55.317 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:55.314+0000 7f78f02c73c0 -1 Falling back to public interface 2023-09-30T00:32:55.317 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:55.314+0000 7f8ffb3ed3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.317 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:55.314+0000 7f8ffb3ed3c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.317 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:55.314+0000 7f8ffb3ed3c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:55.314+0000 7f2b6e1e93c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:55.314+0000 7f2b6e1e93c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:55.314+0000 7f2b6e1e93c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.319 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:55.314+0000 7f9dc8e673c0 -1 Falling back to public interface 2023-09-30T00:32:55.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:55.318+0000 7fe96d1e63c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:55.318+0000 7fe96d1e63c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:55.318+0000 7fe96d1e63c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:55.318+0000 7f2b6e1e93c0 -1 osd.1 0 log_to_monitors true 2023-09-30T00:32:55.322 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:55.318+0000 7f8ffb3ed3c0 -1 osd.4 0 log_to_monitors true 2023-09-30T00:32:55.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:55.322+0000 7f54c2d743c0 -1 Falling back to public interface 2023-09-30T00:32:55.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:55.322+0000 7fe96d1e63c0 -1 osd.0 0 log_to_monitors true 2023-09-30T00:32:55.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:55.322+0000 7f5b3a4ac3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:55.326+0000 7f5b3a4ac3c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:55.326+0000 7f5b3a4ac3c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.334 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:55.330+0000 7f5b3a4ac3c0 -1 osd.3 0 log_to_monitors true 2023-09-30T00:32:55.336 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:55.330+0000 7fbacb63b3c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.336 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:55.334+0000 7fbacb63b3c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.337 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:55.334+0000 7fbacb63b3c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.340 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:55.338+0000 7f78f02c73c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.340 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:55.338+0000 7f78f02c73c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.340 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:55.338+0000 7f78f02c73c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.341 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:55.338+0000 7f9dc8e673c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.341 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:55.338+0000 7f9dc8e673c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.342 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:55.338+0000 7f9dc8e673c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.342 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:55.338+0000 7fbacb63b3c0 -1 osd.7 0 log_to_monitors true 2023-09-30T00:32:55.346 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:55.342+0000 7f9dc8e673c0 -1 osd.5 0 log_to_monitors true 2023-09-30T00:32:55.347 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:55.342+0000 7f78f02c73c0 -1 osd.6 0 log_to_monitors true 2023-09-30T00:32:55.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:55.346+0000 7f54c2d743c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-30T00:32:55.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:55.346+0000 7f54c2d743c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:55.346+0000 7f54c2d743c0 -1 journal do_read_entry(8192): bad header magic 2023-09-30T00:32:55.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:55.350+0000 7f54c2d743c0 -1 osd.2 0 log_to_monitors true 2023-09-30T00:32:55.415 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:32:55.416 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:32:55.204071+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:32:55.433 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:32:55.434 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:32:55.528 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.526+0000 7fc9a9750000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:32:55.584 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.582+0000 7fc9a9750000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:32:55.600 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.594+0000 7f021db34000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:32:55.650 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.646+0000 7fc9a9750000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:32:55.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.654+0000 7f021db34000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:32:55.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.726+0000 7f021db34000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:32:55.809 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.806+0000 7fc9a9750000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:32:55.874 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:55.870+0000 7fc9a9750000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:32:55.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.898+0000 7f021db34000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:32:55.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:55.966+0000 7f021db34000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:32:56.238 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:32:56.411 INFO:teuthology.misc.health.smithi110.stdout: 2023-09-30T00:32:56.411 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:32:56.206072+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:32:56.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:32:56.572 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:56.570+0000 7fc9a9750000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:32:56.629 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:56.626+0000 7fc987031700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:32:56.629 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:56.626+0000 7fc987031700 -1 pg_autoscaler.serve: 2023-09-30T00:32:56.630 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:56.626+0000 7fc987031700 -1 Traceback (most recent call last): 2023-09-30T00:32:56.630 INFO:tasks.ceph.mgr.x.smithi157.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:32:56.631 INFO:tasks.ceph.mgr.x.smithi157.stderr: self._maybe_adjust() 2023-09-30T00:32:56.631 INFO:tasks.ceph.mgr.x.smithi157.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:32:56.631 INFO:tasks.ceph.mgr.x.smithi157.stderr: if self.noautoscale: 2023-09-30T00:32:56.631 INFO:tasks.ceph.mgr.x.smithi157.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:32:56.631 INFO:tasks.ceph.mgr.x.smithi157.stderr: 2023-09-30T00:32:56.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-09-30T00:32:56.722+0000 7f021db34000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:32:57.222 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:57.218+0000 7f54bed37700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:32:57.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:57.218+0000 7f5b3646f700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:32:57.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:57.218+0000 7f2b6a9ad700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:32:57.223 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:57.218+0000 7f9dc4e2a700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T00:32:57.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:57.222+0000 7fe9691a9700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:32:57.227 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:57.222+0000 7f54ae2c8700 -1 osd.2 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:57.222+0000 7f5b25a00700 -1 osd.3 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.228 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:57.226+0000 7f9db43bb700 -1 osd.5 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.228 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:57.226+0000 7f78ec28a700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T00:32:57.229 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:57.226+0000 7fbac75fe700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T00:32:57.229 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:57.226+0000 7f8ff7bb1700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T00:32:57.229 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:57.226+0000 7fe95873a700 -1 osd.0 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:57.222+0000 7f2b5973d700 -1 osd.1 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.231 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:57.226+0000 7fbab6b8f700 -1 osd.7 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.231 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:57.226+0000 7f78db81b700 -1 osd.6 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.231 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:57.226+0000 7f8fe6941700 -1 osd.4 0 handle_osd_map injecting map crc failure 2023-09-30T00:32:57.253 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:57.250+0000 7f9dc13d5700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:57.250+0000 7fe965754700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.257 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:57.254+0000 7fbac3ba9700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:57.258+0000 7f2b66757700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:57.258+0000 7f54bb2e2700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:57.258+0000 7f5b32a1a700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.282 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:57.278+0000 7f8ff395b700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.282 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:57.278+0000 7f78e8835700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:32:57.623 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:57.618+0000 7fc992848700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T00:32:57.623 INFO:tasks.ceph.mgr.x.smithi157.stderr:2023-09-30T00:32:57.618+0000 7fc992848700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T00:32:58.227 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:32:58.222+0000 7f9db43bb700 -1 osd.5 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.227 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:32:58.222+0000 7f5b25a00700 -1 osd.3 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:32:58.222+0000 7f2b5973d700 -1 osd.1 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.228 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:32:58.222+0000 7f54ae2c8700 -1 osd.2 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:32:58.222+0000 7fe95873a700 -1 osd.0 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.229 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:32:58.226+0000 7f8fe6941700 -1 osd.4 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.229 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:32:58.226+0000 7f78db81b700 -1 osd.6 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.229 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:32:58.226+0000 7fbab6b8f700 -1 osd.7 12 handle_osd_map injecting map crc failure 2023-09-30T00:32:58.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2023-09-30T00:32:58.622+0000 7fd4e72ac700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:33:00.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:33:00.026+0000 7f2b57739700 -1 osd.1 13 handle_osd_map injecting map crc failure 2023-09-30T00:33:00.045 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:33:00.042+0000 7fbab4b8b700 -1 osd.7 13 handle_osd_map injecting map crc failure 2023-09-30T00:33:00.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:33:00.054+0000 7f54ac2c4700 -1 osd.2 13 handle_osd_map injecting map crc failure 2023-09-30T00:33:01.277 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:33:01.274+0000 7f9db43bb700 -1 osd.5 14 handle_osd_map injecting map crc failure 2023-09-30T00:33:03.229 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:33:03.517 INFO:teuthology.misc.health.smithi110.stdout: 2023-09-30T00:33:03.517 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":16,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:33:01.265990+0000","last_up_change":"2023-09-30T00:32:58.220653+0000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:32:58.672551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":26915},{"type":"v1","addr":"172.21.15.110:6809","nonce":26915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":26915},{"type":"v1","addr":"172.21.15.110:6811","nonce":26915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":26915},{"type":"v1","addr":"172.21.15.110:6815","nonce":26915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":26915},{"type":"v1","addr":"172.21.15.110:6813","nonce":26915}]},"public_addr":"172.21.15.110:6809/26915","cluster_addr":"172.21.15.110:6811/26915","heartbeat_back_addr":"172.21.15.110:6815/26915","heartbeat_front_addr":"172.21.15.110:6813/26915","state":["exists","up"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":26913},{"type":"v1","addr":"172.21.15.110:6801","nonce":26913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":26913},{"type":"v1","addr":"172.21.15.110:6803","nonce":26913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":26913},{"type":"v1","addr":"172.21.15.110:6807","nonce":26913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":26913},{"type":"v1","addr":"172.21.15.110:6805","nonce":26913}]},"public_addr":"172.21.15.110:6801/26913","cluster_addr":"172.21.15.110:6803/26913","heartbeat_back_addr":"172.21.15.110:6807/26913","heartbeat_front_addr":"172.21.15.110:6805/26913","state":["exists","up"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":26926},{"type":"v1","addr":"172.21.15.110:6825","nonce":26926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":26926},{"type":"v1","addr":"172.21.15.110:6827","nonce":26926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":26926},{"type":"v1","addr":"172.21.15.110:6831","nonce":26926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":26926},{"type":"v1","addr":"172.21.15.110:6829","nonce":26926}]},"public_addr":"172.21.15.110:6825/26926","cluster_addr":"172.21.15.110:6827/26926","heartbeat_back_addr":"172.21.15.110:6831/26926","heartbeat_front_addr":"172.21.15.110:6829/26926","state":["exists","up"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":26914},{"type":"v1","addr":"172.21.15.110:6817","nonce":26914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":26914},{"type":"v1","addr":"172.21.15.110:6819","nonce":26914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":26914},{"type":"v1","addr":"172.21.15.110:6823","nonce":26914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":26914},{"type":"v1","addr":"172.21.15.110:6821","nonce":26914}]},"public_addr":"172.21.15.110:6817/26914","cluster_addr":"172.21.15.110:6819/26914","heartbeat_back_addr":"172.21.15.110:6823/26914","heartbeat_front_addr":"172.21.15.110:6821/26914","state":["exists","up"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":27048},{"type":"v1","addr":"172.21.15.157:6801","nonce":27048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":27048},{"type":"v1","addr":"172.21.15.157:6803","nonce":27048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":27048},{"type":"v1","addr":"172.21.15.157:6807","nonce":27048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":27048},{"type":"v1","addr":"172.21.15.157:6805","nonce":27048}]},"public_addr":"172.21.15.157:6801/27048","cluster_addr":"172.21.15.157:6803/27048","heartbeat_back_addr":"172.21.15.157:6807/27048","heartbeat_front_addr":"172.21.15.157:6805/27048","state":["exists","up"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6823","nonce":27059},{"type":"v1","addr":"172.21.15.157:6825","nonce":27059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":27059},{"type":"v1","addr":"172.21.15.157:6827","nonce":27059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":27059},{"type":"v1","addr":"172.21.15.157:6831","nonce":27059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":27059},{"type":"v1","addr":"172.21.15.157:6829","nonce":27059}]},"public_addr":"172.21.15.157:6825/27059","cluster_addr":"172.21.15.157:6827/27059","heartbeat_back_addr":"172.21.15.157:6831/27059","heartbeat_front_addr":"172.21.15.157:6829/27059","state":["exists","up"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":27049},{"type":"v1","addr":"172.21.15.157:6817","nonce":27049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":27049},{"type":"v1","addr":"172.21.15.157:6819","nonce":27049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":27049},{"type":"v1","addr":"172.21.15.157:6824","nonce":27049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":27049},{"type":"v1","addr":"172.21.15.157:6821","nonce":27049}]},"public_addr":"172.21.15.157:6817/27049","cluster_addr":"172.21.15.157:6819/27049","heartbeat_back_addr":"172.21.15.157:6824/27049","heartbeat_front_addr":"172.21.15.157:6821/27049","state":["exists","up"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":27047},{"type":"v1","addr":"172.21.15.157:6809","nonce":27047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":27047},{"type":"v1","addr":"172.21.15.157:6811","nonce":27047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":27047},{"type":"v1","addr":"172.21.15.157:6815","nonce":27047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":27047},{"type":"v1","addr":"172.21.15.157:6813","nonce":27047}]},"public_addr":"172.21.15.157:6809/27047","cluster_addr":"172.21.15.157:6811/27047","heartbeat_back_addr":"172.21.15.157:6815/27047","heartbeat_front_addr":"172.21.15.157:6813/27047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.311002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.341137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.385183+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.343146+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.357149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.346472+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.396435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.305942+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:33:03.527 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T00:33:03.527 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:33:03.527 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:33:04.303 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2023-09-30T00:33:04.314 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:33:05.296 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:33:05.294+0000 7f9db43bb700 -1 osd.5 17 handle_osd_map injecting map crc failure 2023-09-30T00:33:05.311 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:33:05.322 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:33:05.322 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:33:05.322 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:33:05.323 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:33:05.323 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:33:05.654 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:05.669 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":26741},{"type":"v1","addr":"172.21.15.157:6833","nonce":26741}]},"active_addr":"172.21.15.157:6833/26741","active_change":"2023-09-30T00:32:56.592693+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4103,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":4126736753}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":3760821151}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":2014741634}]}]} 2023-09-30T00:33:05.675 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:33:05.675 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:33:05.676 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:33:05.964 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:05.964 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":18,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:33:05.290072+0000","last_up_change":"2023-09-30T00:32:58.220653+0000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:32:58.672551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:33:03.847465+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":26915},{"type":"v1","addr":"172.21.15.110:6809","nonce":26915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":26915},{"type":"v1","addr":"172.21.15.110:6811","nonce":26915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":26915},{"type":"v1","addr":"172.21.15.110:6815","nonce":26915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":26915},{"type":"v1","addr":"172.21.15.110:6813","nonce":26915}]},"public_addr":"172.21.15.110:6809/26915","cluster_addr":"172.21.15.110:6811/26915","heartbeat_back_addr":"172.21.15.110:6815/26915","heartbeat_front_addr":"172.21.15.110:6813/26915","state":["exists","up"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":26913},{"type":"v1","addr":"172.21.15.110:6801","nonce":26913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":26913},{"type":"v1","addr":"172.21.15.110:6803","nonce":26913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":26913},{"type":"v1","addr":"172.21.15.110:6807","nonce":26913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":26913},{"type":"v1","addr":"172.21.15.110:6805","nonce":26913}]},"public_addr":"172.21.15.110:6801/26913","cluster_addr":"172.21.15.110:6803/26913","heartbeat_back_addr":"172.21.15.110:6807/26913","heartbeat_front_addr":"172.21.15.110:6805/26913","state":["exists","up"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":26926},{"type":"v1","addr":"172.21.15.110:6825","nonce":26926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":26926},{"type":"v1","addr":"172.21.15.110:6827","nonce":26926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":26926},{"type":"v1","addr":"172.21.15.110:6831","nonce":26926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":26926},{"type":"v1","addr":"172.21.15.110:6829","nonce":26926}]},"public_addr":"172.21.15.110:6825/26926","cluster_addr":"172.21.15.110:6827/26926","heartbeat_back_addr":"172.21.15.110:6831/26926","heartbeat_front_addr":"172.21.15.110:6829/26926","state":["exists","up"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":26914},{"type":"v1","addr":"172.21.15.110:6817","nonce":26914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":26914},{"type":"v1","addr":"172.21.15.110:6819","nonce":26914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":26914},{"type":"v1","addr":"172.21.15.110:6823","nonce":26914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":26914},{"type":"v1","addr":"172.21.15.110:6821","nonce":26914}]},"public_addr":"172.21.15.110:6817/26914","cluster_addr":"172.21.15.110:6819/26914","heartbeat_back_addr":"172.21.15.110:6823/26914","heartbeat_front_addr":"172.21.15.110:6821/26914","state":["exists","up"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":27048},{"type":"v1","addr":"172.21.15.157:6801","nonce":27048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":27048},{"type":"v1","addr":"172.21.15.157:6803","nonce":27048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":27048},{"type":"v1","addr":"172.21.15.157:6807","nonce":27048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":27048},{"type":"v1","addr":"172.21.15.157:6805","nonce":27048}]},"public_addr":"172.21.15.157:6801/27048","cluster_addr":"172.21.15.157:6803/27048","heartbeat_back_addr":"172.21.15.157:6807/27048","heartbeat_front_addr":"172.21.15.157:6805/27048","state":["exists","up"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6823","nonce":27059},{"type":"v1","addr":"172.21.15.157:6825","nonce":27059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":27059},{"type":"v1","addr":"172.21.15.157:6827","nonce":27059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":27059},{"type":"v1","addr":"172.21.15.157:6831","nonce":27059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":27059},{"type":"v1","addr":"172.21.15.157:6829","nonce":27059}]},"public_addr":"172.21.15.157:6825/27059","cluster_addr":"172.21.15.157:6827/27059","heartbeat_back_addr":"172.21.15.157:6831/27059","heartbeat_front_addr":"172.21.15.157:6829/27059","state":["exists","up"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":27049},{"type":"v1","addr":"172.21.15.157:6817","nonce":27049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":27049},{"type":"v1","addr":"172.21.15.157:6819","nonce":27049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":27049},{"type":"v1","addr":"172.21.15.157:6824","nonce":27049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":27049},{"type":"v1","addr":"172.21.15.157:6821","nonce":27049}]},"public_addr":"172.21.15.157:6817/27049","cluster_addr":"172.21.15.157:6819/27049","heartbeat_back_addr":"172.21.15.157:6824/27049","heartbeat_front_addr":"172.21.15.157:6821/27049","state":["exists","up"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":27047},{"type":"v1","addr":"172.21.15.157:6809","nonce":27047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":27047},{"type":"v1","addr":"172.21.15.157:6811","nonce":27047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":27047},{"type":"v1","addr":"172.21.15.157:6815","nonce":27047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":27047},{"type":"v1","addr":"172.21.15.157:6813","nonce":27047}]},"public_addr":"172.21.15.157:6809/27047","cluster_addr":"172.21.15.157:6811/27047","heartbeat_back_addr":"172.21.15.157:6815/27047","heartbeat_front_addr":"172.21.15.157:6813/27047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.311002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.341137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.385183+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.343146+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.357149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.346472+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.396435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.305942+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:33:05.978 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:33:05.978 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:33:06.262 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:06.263 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":18,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:33:05.290072+0000","last_up_change":"2023-09-30T00:32:58.220653+0000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:32:58.672551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:33:03.847465+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":26915},{"type":"v1","addr":"172.21.15.110:6809","nonce":26915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":26915},{"type":"v1","addr":"172.21.15.110:6811","nonce":26915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":26915},{"type":"v1","addr":"172.21.15.110:6815","nonce":26915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":26915},{"type":"v1","addr":"172.21.15.110:6813","nonce":26915}]},"public_addr":"172.21.15.110:6809/26915","cluster_addr":"172.21.15.110:6811/26915","heartbeat_back_addr":"172.21.15.110:6815/26915","heartbeat_front_addr":"172.21.15.110:6813/26915","state":["exists","up"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":26913},{"type":"v1","addr":"172.21.15.110:6801","nonce":26913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":26913},{"type":"v1","addr":"172.21.15.110:6803","nonce":26913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":26913},{"type":"v1","addr":"172.21.15.110:6807","nonce":26913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":26913},{"type":"v1","addr":"172.21.15.110:6805","nonce":26913}]},"public_addr":"172.21.15.110:6801/26913","cluster_addr":"172.21.15.110:6803/26913","heartbeat_back_addr":"172.21.15.110:6807/26913","heartbeat_front_addr":"172.21.15.110:6805/26913","state":["exists","up"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":26926},{"type":"v1","addr":"172.21.15.110:6825","nonce":26926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":26926},{"type":"v1","addr":"172.21.15.110:6827","nonce":26926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":26926},{"type":"v1","addr":"172.21.15.110:6831","nonce":26926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":26926},{"type":"v1","addr":"172.21.15.110:6829","nonce":26926}]},"public_addr":"172.21.15.110:6825/26926","cluster_addr":"172.21.15.110:6827/26926","heartbeat_back_addr":"172.21.15.110:6831/26926","heartbeat_front_addr":"172.21.15.110:6829/26926","state":["exists","up"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":26914},{"type":"v1","addr":"172.21.15.110:6817","nonce":26914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":26914},{"type":"v1","addr":"172.21.15.110:6819","nonce":26914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":26914},{"type":"v1","addr":"172.21.15.110:6823","nonce":26914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":26914},{"type":"v1","addr":"172.21.15.110:6821","nonce":26914}]},"public_addr":"172.21.15.110:6817/26914","cluster_addr":"172.21.15.110:6819/26914","heartbeat_back_addr":"172.21.15.110:6823/26914","heartbeat_front_addr":"172.21.15.110:6821/26914","state":["exists","up"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":27048},{"type":"v1","addr":"172.21.15.157:6801","nonce":27048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":27048},{"type":"v1","addr":"172.21.15.157:6803","nonce":27048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":27048},{"type":"v1","addr":"172.21.15.157:6807","nonce":27048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":27048},{"type":"v1","addr":"172.21.15.157:6805","nonce":27048}]},"public_addr":"172.21.15.157:6801/27048","cluster_addr":"172.21.15.157:6803/27048","heartbeat_back_addr":"172.21.15.157:6807/27048","heartbeat_front_addr":"172.21.15.157:6805/27048","state":["exists","up"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6823","nonce":27059},{"type":"v1","addr":"172.21.15.157:6825","nonce":27059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":27059},{"type":"v1","addr":"172.21.15.157:6827","nonce":27059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":27059},{"type":"v1","addr":"172.21.15.157:6831","nonce":27059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":27059},{"type":"v1","addr":"172.21.15.157:6829","nonce":27059}]},"public_addr":"172.21.15.157:6825/27059","cluster_addr":"172.21.15.157:6827/27059","heartbeat_back_addr":"172.21.15.157:6831/27059","heartbeat_front_addr":"172.21.15.157:6829/27059","state":["exists","up"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":27049},{"type":"v1","addr":"172.21.15.157:6817","nonce":27049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":27049},{"type":"v1","addr":"172.21.15.157:6819","nonce":27049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":27049},{"type":"v1","addr":"172.21.15.157:6824","nonce":27049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":27049},{"type":"v1","addr":"172.21.15.157:6821","nonce":27049}]},"public_addr":"172.21.15.157:6817/27049","cluster_addr":"172.21.15.157:6819/27049","heartbeat_back_addr":"172.21.15.157:6824/27049","heartbeat_front_addr":"172.21.15.157:6821/27049","state":["exists","up"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":27047},{"type":"v1","addr":"172.21.15.157:6809","nonce":27047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":27047},{"type":"v1","addr":"172.21.15.157:6811","nonce":27047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":27047},{"type":"v1","addr":"172.21.15.157:6815","nonce":27047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":27047},{"type":"v1","addr":"172.21.15.157:6813","nonce":27047}]},"public_addr":"172.21.15.157:6809/27047","cluster_addr":"172.21.15.157:6811/27047","heartbeat_back_addr":"172.21.15.157:6815/27047","heartbeat_front_addr":"172.21.15.157:6813/27047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.311002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.341137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.385183+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.343146+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.357149+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.346472+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.396435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:32:56.305942+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:33:06.277 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:33:06.278 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:33:06.278 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:33:06.278 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:33:06.279 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T00:33:06.279 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T00:33:06.279 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T00:33:06.280 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T00:33:06.479 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.480 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:33:06.489 INFO:teuthology.orchestra.run.smithi110.stdout:55834574852 2023-09-30T00:33:06.490 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:33:06.493 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.493 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:33:06.498 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:33:06.500 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.500 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:33:06.501 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.502 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:33:06.508 INFO:teuthology.orchestra.run.smithi110.stdout:55834574852 2023-09-30T00:33:06.508 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:33:06.511 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.511 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:33:06.929 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:33:06.926+0000 7f9db23b7700 -1 osd.5 18 handle_osd_map injecting map crc failure 2023-09-30T00:33:06.943 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2023-09-30T00:33:06.948 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:06.950 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2023-09-30T00:33:06.963 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2023-09-30T00:33:06.964 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.7 2023-09-30T00:33:06.964 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2023-09-30T00:33:06.969 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.1 2023-09-30T00:33:06.975 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2023-09-30T00:33:06.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.2 2023-09-30T00:33:06.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2023-09-30T00:33:06.982 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.5 2023-09-30T00:33:06.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.3 2023-09-30T00:33:06.995 INFO:teuthology.orchestra.run.smithi110.stdout:55834574850 2023-09-30T00:33:07.005 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:07.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.0 2023-09-30T00:33:07.017 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2023-09-30T00:33:07.965 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:33:07.970 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:33:07.979 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:33:07.980 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:33:07.982 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:33:07.989 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:33:08.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:33:08.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:33:08.424 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:08.443 INFO:teuthology.orchestra.run.smithi110.stdout:55834574852 2023-09-30T00:33:08.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.7 2023-09-30T00:33:08.452 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2023-09-30T00:33:08.457 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.462 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:08.467 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:08.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2023-09-30T00:33:08.481 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2023-09-30T00:33:08.481 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.485 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:08.489 INFO:teuthology.orchestra.run.smithi110.stdout:55834574852 2023-09-30T00:33:08.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2023-09-30T00:33:08.499 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.500 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:08.504 INFO:teuthology.orchestra.run.smithi110.stdout:55834574851 2023-09-30T00:33:08.505 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2023-09-30T00:33:08.505 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2023-09-30T00:33:08.513 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2023-09-30T00:33:08.516 DEBUG:teuthology.parallel:result is None 2023-09-30T00:33:08.516 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:33:08.516 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:33:08.828 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:08.830 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:33:08.841 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-09-30T00:33:08.608149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6368276,"kb_used_data":743227436,"kb_used_omap":442,"kb_used_meta":0,"kb_avail":743226764,"statfs":{"total":767585320960,"available":761064206336,"internally_reserved":688128,"allocated":761064894464,"data_stored":761064894464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":453067,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":29,"num_write":40,"num_write_kb":717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.315913"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.407177+0000","last_change":"2023-09-30T00:33:05.407177+0000","last_active":"2023-09-30T00:33:05.407177+0000","last_peered":"2023-09-30T00:33:05.407177+0000","last_clean":"2023-09-30T00:33:05.407177+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:33:05.407177+0000","last_undegraded":"2023-09-30T00:33:05.407177+0000","last_fullsized":"2023-09-30T00:33:05.407177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:11.335052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.304730+0000","last_change":"2023-09-30T00:33:05.304021+0000","last_active":"2023-09-30T00:33:05.304730+0000","last_peered":"2023-09-30T00:33:05.304730+0000","last_clean":"2023-09-30T00:33:05.304730+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:33:05.304730+0000","last_undegraded":"2023-09-30T00:33:05.304730+0000","last_fullsized":"2023-09-30T00:33:05.304730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:32.465376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.306861+0000","last_change":"2023-09-30T00:33:05.306686+0000","last_active":"2023-09-30T00:33:05.306861+0000","last_peered":"2023-09-30T00:33:05.306861+0000","last_clean":"2023-09-30T00:33:05.306861+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:33:05.306861+0000","last_undegraded":"2023-09-30T00:33:05.306861+0000","last_fullsized":"2023-09-30T00:33:05.306861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:17.386366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.307372+0000","last_change":"2023-09-30T00:33:05.307180+0000","last_active":"2023-09-30T00:33:05.307372+0000","last_peered":"2023-09-30T00:33:05.307372+0000","last_clean":"2023-09-30T00:33:05.307372+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:33:05.307372+0000","last_undegraded":"2023-09-30T00:33:05.307372+0000","last_fullsized":"2023-09-30T00:33:05.307372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:06.406849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.305199+0000","last_change":"2023-09-30T00:33:05.305199+0000","last_active":"2023-09-30T00:33:05.305199+0000","last_peered":"2023-09-30T00:33:05.305199+0000","last_clean":"2023-09-30T00:33:05.305199+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:33:05.305199+0000","last_undegraded":"2023-09-30T00:33:05.305199+0000","last_fullsized":"2023-09-30T00:33:05.305199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:30.688222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.430947+0000","last_change":"2023-09-30T00:33:05.430947+0000","last_active":"2023-09-30T00:33:05.430947+0000","last_peered":"2023-09-30T00:33:05.430947+0000","last_clean":"2023-09-30T00:33:05.430947+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:33:05.430947+0000","last_undegraded":"2023-09-30T00:33:05.430947+0000","last_fullsized":"2023-09-30T00:33:05.430947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:30.688222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.801696+0000","last_change":"2023-09-30T00:33:05.701245+0000","last_active":"2023-09-30T00:33:05.801696+0000","last_peered":"2023-09-30T00:33:05.801696+0000","last_clean":"2023-09-30T00:33:05.801696+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:33:05.801696+0000","last_undegraded":"2023-09-30T00:33:05.801696+0000","last_fullsized":"2023-09-30T00:33:05.801696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:10.950593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.304273+0000","last_change":"2023-09-30T00:33:05.304273+0000","last_active":"2023-09-30T00:33:05.304273+0000","last_peered":"2023-09-30T00:33:05.304273+0000","last_clean":"2023-09-30T00:33:05.304273+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:33:05.304273+0000","last_undegraded":"2023-09-30T00:33:05.304273+0000","last_fullsized":"2023-09-30T00:33:05.304273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:19.360054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'60","reported_seq":93,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-30T00:33:06.421120+0000","last_change":"2023-09-30T00:33:00.378988+0000","last_active":"2023-09-30T00:33:06.421120+0000","last_peered":"2023-09-30T00:33:06.421120+0000","last_clean":"2023-09-30T00:33:06.421120+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:33:06.421120+0000","last_undegraded":"2023-09-30T00:33:06.421120+0000","last_fullsized":"2023-09-30T00:33:06.421120+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_clean_scrub_stamp":"2023-09-30T00:32:59.239112+0000","objects_scrubbed":0,"log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:18.969410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796592,"kb_used_data":92902908,"kb_used_omap":89,"kb_used_meta":0,"kb_avail":92902788,"statfs":{"total":95948165120,"available":95132454912,"internally_reserved":122880,"allocated":95132577792,"data_stored":95132577792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91506,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795780,"kb_used_data":92903608,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903600,"statfs":{"total":95948165120,"available":95133286400,"internally_reserved":8192,"allocated":95133294592,"data_stored":95133294592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48191,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795764,"kb_used_data":92903616,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903616,"statfs":{"total":95948165120,"available":95133302784,"internally_reserved":0,"allocated":95133302784,"data_stored":95133302784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43771,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795700,"kb_used_data":92903688,"kb_used_omap":36,"kb_used_meta":0,"kb_avail":92903680,"statfs":{"total":95948165120,"available":95133368320,"internally_reserved":8192,"allocated":95133376512,"data_stored":95133376512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37718,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795784,"kb_used_data":92903612,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":92903596,"statfs":{"total":95948165120,"available":95133282304,"internally_reserved":16384,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42904,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903612,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":8192,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43746,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5780000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903560,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":28672,"allocated":95133245440,"data_stored":95133245440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58854,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797032,"kb_used_data":92902832,"kb_used_omap":84,"kb_used_meta":0,"kb_avail":92902348,"statfs":{"total":95948165120,"available":95132004352,"internally_reserved":495616,"allocated":95132499968,"data_stored":95132499968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86377,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]}]}],"pool_statfs":[]}} 2023-09-30T00:33:08.842 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:33:09.104 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:09.106 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:33:09.115 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-09-30T00:33:08.608149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6368276,"kb_used_data":743227436,"kb_used_omap":442,"kb_used_meta":0,"kb_avail":743226764,"statfs":{"total":767585320960,"available":761064206336,"internally_reserved":688128,"allocated":761064894464,"data_stored":761064894464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":453067,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":29,"num_write":40,"num_write_kb":717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.315913"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.407177+0000","last_change":"2023-09-30T00:33:05.407177+0000","last_active":"2023-09-30T00:33:05.407177+0000","last_peered":"2023-09-30T00:33:05.407177+0000","last_clean":"2023-09-30T00:33:05.407177+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:33:05.407177+0000","last_undegraded":"2023-09-30T00:33:05.407177+0000","last_fullsized":"2023-09-30T00:33:05.407177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:11.335052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.304730+0000","last_change":"2023-09-30T00:33:05.304021+0000","last_active":"2023-09-30T00:33:05.304730+0000","last_peered":"2023-09-30T00:33:05.304730+0000","last_clean":"2023-09-30T00:33:05.304730+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:33:05.304730+0000","last_undegraded":"2023-09-30T00:33:05.304730+0000","last_fullsized":"2023-09-30T00:33:05.304730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:32.465376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.306861+0000","last_change":"2023-09-30T00:33:05.306686+0000","last_active":"2023-09-30T00:33:05.306861+0000","last_peered":"2023-09-30T00:33:05.306861+0000","last_clean":"2023-09-30T00:33:05.306861+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:33:05.306861+0000","last_undegraded":"2023-09-30T00:33:05.306861+0000","last_fullsized":"2023-09-30T00:33:05.306861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:17.386366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.307372+0000","last_change":"2023-09-30T00:33:05.307180+0000","last_active":"2023-09-30T00:33:05.307372+0000","last_peered":"2023-09-30T00:33:05.307372+0000","last_clean":"2023-09-30T00:33:05.307372+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:33:05.307372+0000","last_undegraded":"2023-09-30T00:33:05.307372+0000","last_fullsized":"2023-09-30T00:33:05.307372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:06.406849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.305199+0000","last_change":"2023-09-30T00:33:05.305199+0000","last_active":"2023-09-30T00:33:05.305199+0000","last_peered":"2023-09-30T00:33:05.305199+0000","last_clean":"2023-09-30T00:33:05.305199+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:33:05.305199+0000","last_undegraded":"2023-09-30T00:33:05.305199+0000","last_fullsized":"2023-09-30T00:33:05.305199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:30.688222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.430947+0000","last_change":"2023-09-30T00:33:05.430947+0000","last_active":"2023-09-30T00:33:05.430947+0000","last_peered":"2023-09-30T00:33:05.430947+0000","last_clean":"2023-09-30T00:33:05.430947+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:33:05.430947+0000","last_undegraded":"2023-09-30T00:33:05.430947+0000","last_fullsized":"2023-09-30T00:33:05.430947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:30.688222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.801696+0000","last_change":"2023-09-30T00:33:05.701245+0000","last_active":"2023-09-30T00:33:05.801696+0000","last_peered":"2023-09-30T00:33:05.801696+0000","last_clean":"2023-09-30T00:33:05.801696+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:33:05.801696+0000","last_undegraded":"2023-09-30T00:33:05.801696+0000","last_fullsized":"2023-09-30T00:33:05.801696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:10.950593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:33:05.304273+0000","last_change":"2023-09-30T00:33:05.304273+0000","last_active":"2023-09-30T00:33:05.304273+0000","last_peered":"2023-09-30T00:33:05.304273+0000","last_clean":"2023-09-30T00:33:05.304273+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:33:05.304273+0000","last_undegraded":"2023-09-30T00:33:05.304273+0000","last_fullsized":"2023-09-30T00:33:05.304273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:33:04.279938+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:19.360054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'60","reported_seq":93,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-30T00:33:06.421120+0000","last_change":"2023-09-30T00:33:00.378988+0000","last_active":"2023-09-30T00:33:06.421120+0000","last_peered":"2023-09-30T00:33:06.421120+0000","last_clean":"2023-09-30T00:33:06.421120+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:33:06.421120+0000","last_undegraded":"2023-09-30T00:33:06.421120+0000","last_fullsized":"2023-09-30T00:33:06.421120+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_clean_scrub_stamp":"2023-09-30T00:32:59.239112+0000","objects_scrubbed":0,"log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:34:18.969410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":92,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796592,"kb_used_data":92902908,"kb_used_omap":89,"kb_used_meta":0,"kb_avail":92902788,"statfs":{"total":95948165120,"available":95132454912,"internally_reserved":122880,"allocated":95132577792,"data_stored":95132577792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91506,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795780,"kb_used_data":92903608,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903600,"statfs":{"total":95948165120,"available":95133286400,"internally_reserved":8192,"allocated":95133294592,"data_stored":95133294592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48191,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795764,"kb_used_data":92903616,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903616,"statfs":{"total":95948165120,"available":95133302784,"internally_reserved":0,"allocated":95133302784,"data_stored":95133302784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43771,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795700,"kb_used_data":92903688,"kb_used_omap":36,"kb_used_meta":0,"kb_avail":92903680,"statfs":{"total":95948165120,"available":95133368320,"internally_reserved":8192,"allocated":95133376512,"data_stored":95133376512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37718,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795784,"kb_used_data":92903612,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":92903596,"statfs":{"total":95948165120,"available":95133282304,"internally_reserved":16384,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42904,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795776,"kb_used_data":92903612,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903604,"statfs":{"total":95948165120,"available":95133290496,"internally_reserved":8192,"allocated":95133298688,"data_stored":95133298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43746,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5780000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903560,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":28672,"allocated":95133245440,"data_stored":95133245440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58854,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797032,"kb_used_data":92902832,"kb_used_omap":84,"kb_used_meta":0,"kb_avail":92902348,"statfs":{"total":95948165120,"available":95132004352,"internally_reserved":495616,"allocated":95132499968,"data_stored":95132499968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86377,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]}]}],"pool_statfs":[]}} 2023-09-30T00:33:09.117 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:33:09.117 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:33:09.117 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:33:09.117 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:09.401 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:09.401 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:12.414 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:12.739 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:12.739 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:15.754 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:16.077 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:16.078 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:19.092 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:19.419 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:19.419 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:22.433 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:22.762 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:22.762 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:25.775 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:26.096 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:26.096 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:29.111 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:29.440 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:29.440 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:32.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:32.782 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:32.782 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:35.802 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:36.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:36.128 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:39.147 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:39.472 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:39.472 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:42.490 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:42.809 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:42.810 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:45.823 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:46.149 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:46.149 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:49.162 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:49.483 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:49.483 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:52.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:52.813 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:52.813 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:55.827 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:56.157 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:56.157 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:33:59.175 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:33:59.501 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:33:59.501 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:01.145 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:34:01.141+0000 7f8fe493d700 -1 osd.4 20 handle_osd_map injecting map crc failure 2023-09-30T00:34:01.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:34:01.258+0000 7f54ae2c8700 -1 osd.2 21 handle_osd_map injecting map crc failure 2023-09-30T00:34:02.517 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:02.839 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:02.840 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:05.853 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:06.174 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:06.174 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:08.665 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:34:08.661+0000 7f9db23b7700 -1 osd.5 22 handle_osd_map injecting map crc failure 2023-09-30T00:34:09.045 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:34:09.041+0000 7fbab4b8b700 -1 osd.7 22 handle_osd_map injecting map crc failure 2023-09-30T00:34:09.046 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:34:09.041+0000 7f78d9817700 -1 osd.6 22 handle_osd_map injecting map crc failure 2023-09-30T00:34:09.188 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:09.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:34:09.302+0000 7fe95873a700 -1 osd.0 23 handle_osd_map injecting map crc failure 2023-09-30T00:34:09.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:09.511 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:10.266 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:34:10.261+0000 7f9db23b7700 -1 osd.5 23 handle_osd_map injecting map crc failure 2023-09-30T00:34:10.446 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:34:10.441+0000 7f78d9817700 -1 osd.6 23 handle_osd_map injecting map crc failure 2023-09-30T00:34:12.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:12.849 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:12.849 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:13.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:34:13.326+0000 7fe95873a700 -1 osd.0 24 handle_osd_map injecting map crc failure 2023-09-30T00:34:15.863 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:16.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:16.186 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:19.200 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:19.525 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:19.525 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:22.541 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:22.872 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:22.872 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:23.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:34:23.766+0000 7f2b57739700 -1 osd.1 26 handle_osd_map injecting map crc failure 2023-09-30T00:34:23.769 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:34:23.765+0000 7fbab4b8b700 -1 osd.7 26 handle_osd_map injecting map crc failure 2023-09-30T00:34:25.887 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:26.211 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:26.211 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:29.226 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:29.548 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:29.549 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:32.563 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:32.885 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:32.885 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:35.899 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:36.223 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:36.223 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:39.236 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:39.562 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:39.563 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:42.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:42.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:42.906 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:45.924 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:46.254 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:46.254 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:49.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:49.603 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:49.603 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:52.617 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:52.943 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:52.943 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:55.957 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:56.281 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:56.281 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:34:59.296 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:34:59.498 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-09-30T00:34:59.494+0000 7fe95873a700 -1 osd.0 27 handle_osd_map injecting map crc failure 2023-09-30T00:34:59.622 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:34:59.622 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:02.635 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:02.966 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:02.966 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:05.980 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:06.312 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:06.313 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:09.327 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:09.648 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:09.648 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:12.662 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:12.989 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:12.989 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:16.003 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:16.337 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:16.338 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:17.360 INFO:tasks.ceph.osd.6.smithi157.stderr:2023-09-30T00:35:17.357+0000 7f78d9817700 -1 osd.6 28 handle_osd_map injecting map crc failure 2023-09-30T00:35:19.352 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:19.669 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:19.669 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:22.684 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:23.002 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:23.003 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:26.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:26.348 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:26.348 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:26.352 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:35:26.349+0000 7f8fe6941700 -1 osd.4 32 handle_osd_map injecting map crc failure 2023-09-30T00:35:29.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:29.689 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:29.689 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:32.704 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:33.030 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:33.030 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:36.045 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:36.360 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:36.361 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:39.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:39.694 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:39.694 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:42.708 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:43.044 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:43.045 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:46.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:46.383 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:46.383 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:49.398 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:49.725 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:49.725 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:52.739 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:53.065 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:53.066 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:56.080 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:56.406 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:56.406 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:35:59.420 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:35:59.742 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:35:59.742 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:02.757 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:03.086 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:03.086 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:06.100 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:06.420 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:06.420 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:09.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:09.757 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:09.757 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:12.772 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:13.093 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:13.093 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:16.113 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:16.444 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:16.445 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:19.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:19.784 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:19.785 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:22.798 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:23.120 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:23.120 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:26.134 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:26.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:26.468 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:29.489 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:29.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:36:29.496+0000 7f54ae2c8700 -1 osd.2 36 handle_osd_map injecting map crc failure 2023-09-30T00:36:29.810 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:29.810 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:32.825 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:33.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:33.147 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:33.878 INFO:tasks.ceph.osd.7.smithi157.stderr:2023-09-30T00:36:33.875+0000 7fbab4b8b700 -1 osd.7 37 handle_osd_map injecting map crc failure 2023-09-30T00:36:36.162 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:36.488 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:36.489 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:39.503 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:39.827 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:39.827 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:42.848 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:43.172 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:43.173 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:44.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:36:44.500+0000 7f5b239fc700 -1 osd.3 40 handle_osd_map injecting map crc failure 2023-09-30T00:36:46.187 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:46.518 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:46.519 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:48.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-09-30T00:36:48.504+0000 7f54ac2c4700 -1 osd.2 41 handle_osd_map injecting map crc failure 2023-09-30T00:36:49.533 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:49.853 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:49.853 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:52.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:53.194 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:53.194 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:56.208 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:56.536 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:56.536 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:36:59.556 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:36:59.886 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:36:59.886 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:02.901 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:03.220 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:03.220 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:06.235 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:06.558 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:06.558 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:09.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:09.902 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:09.902 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:12.916 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:13.252 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:13.252 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:16.272 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:16.595 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:16.595 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:19.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:19.951 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:19.951 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:22.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:23.293 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:23.294 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:26.308 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:26.639 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:26.639 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:29.654 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:29.973 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:29.973 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:32.987 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:33.308 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:33.308 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:35.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:37:35.527+0000 7f2b5973d700 -1 osd.1 43 handle_osd_map injecting map crc failure 2023-09-30T00:37:36.020 INFO:tasks.ceph.osd.5.smithi157.stderr:2023-09-30T00:37:36.018+0000 7f9db23b7700 -1 osd.5 43 handle_osd_map injecting map crc failure 2023-09-30T00:37:36.322 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:36.644 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:36.645 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:39.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:39.980 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:39.980 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:42.995 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:43.319 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:43.319 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:46.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:46.658 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:46.658 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:49.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:49.995 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:49.995 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:53.009 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:53.337 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:53.337 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:56.352 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:37:56.678 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:37:56.679 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:37:59.692 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:38:00.024 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:00.024 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:38:03.039 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:38:03.368 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:03.368 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:38:06.382 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:38:06.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-09-30T00:38:06.402+0000 7f2b5973d700 -1 osd.1 47 handle_osd_map injecting map crc failure 2023-09-30T00:38:06.713 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:06.713 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:38:07.425 INFO:tasks.ceph.osd.4.smithi157.stderr:2023-09-30T00:38:07.421+0000 7f8fe6941700 -1 osd.4 48 handle_osd_map injecting map crc failure 2023-09-30T00:38:09.731 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:38:10.060 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:10.060 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:38:10.074 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:38:10.074 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:38:10.356 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":49,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:38:07.413214+0000","last_up_change":"2023-09-30T00:32:58.220653+0000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:32:58.672551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:33:03.847465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":26915},{"type":"v1","addr":"172.21.15.110:6809","nonce":26915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":26915},{"type":"v1","addr":"172.21.15.110:6811","nonce":26915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":26915},{"type":"v1","addr":"172.21.15.110:6815","nonce":26915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":26915},{"type":"v1","addr":"172.21.15.110:6813","nonce":26915}]},"public_addr":"172.21.15.110:6809/26915","cluster_addr":"172.21.15.110:6811/26915","heartbeat_back_addr":"172.21.15.110:6815/26915","heartbeat_front_addr":"172.21.15.110:6813/26915","state":["exists","up"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":26913},{"type":"v1","addr":"172.21.15.110:6801","nonce":26913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":26913},{"type":"v1","addr":"172.21.15.110:6803","nonce":26913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":26913},{"type":"v1","addr":"172.21.15.110:6807","nonce":26913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":26913},{"type":"v1","addr":"172.21.15.110:6805","nonce":26913}]},"public_addr":"172.21.15.110:6801/26913","cluster_addr":"172.21.15.110:6803/26913","heartbeat_back_addr":"172.21.15.110:6807/26913","heartbeat_front_addr":"172.21.15.110:6805/26913","state":["exists","up"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":26926},{"type":"v1","addr":"172.21.15.110:6825","nonce":26926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":26926},{"type":"v1","addr":"172.21.15.110:6827","nonce":26926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":26926},{"type":"v1","addr":"172.21.15.110:6831","nonce":26926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":26926},{"type":"v1","addr":"172.21.15.110:6829","nonce":26926}]},"public_addr":"172.21.15.110:6825/26926","cluster_addr":"172.21.15.110:6827/26926","heartbeat_back_addr":"172.21.15.110:6831/26926","heartbeat_front_addr":"172.21.15.110:6829/26926","state":["exists","up"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":26914},{"type":"v1","addr":"172.21.15.110:6817","nonce":26914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":26914},{"type":"v1","addr":"172.21.15.110:6819","nonce":26914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":26914},{"type":"v1","addr":"172.21.15.110:6823","nonce":26914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":26914},{"type":"v1","addr":"172.21.15.110:6821","nonce":26914}]},"public_addr":"172.21.15.110:6817/26914","cluster_addr":"172.21.15.110:6819/26914","heartbeat_back_addr":"172.21.15.110:6823/26914","heartbeat_front_addr":"172.21.15.110:6821/26914","state":["exists","up"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":27048},{"type":"v1","addr":"172.21.15.157:6801","nonce":27048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":27048},{"type":"v1","addr":"172.21.15.157:6803","nonce":27048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":27048},{"type":"v1","addr":"172.21.15.157:6807","nonce":27048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":27048},{"type":"v1","addr":"172.21.15.157:6805","nonce":27048}]},"public_addr":"172.21.15.157:6801/27048","cluster_addr":"172.21.15.157:6803/27048","heartbeat_back_addr":"172.21.15.157:6807/27048","heartbeat_front_addr":"172.21.15.157:6805/27048","state":["exists","up"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6823","nonce":27059},{"type":"v1","addr":"172.21.15.157:6825","nonce":27059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":27059},{"type":"v1","addr":"172.21.15.157:6827","nonce":27059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":27059},{"type":"v1","addr":"172.21.15.157:6831","nonce":27059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":27059},{"type":"v1","addr":"172.21.15.157:6829","nonce":27059}]},"public_addr":"172.21.15.157:6825/27059","cluster_addr":"172.21.15.157:6827/27059","heartbeat_back_addr":"172.21.15.157:6831/27059","heartbeat_front_addr":"172.21.15.157:6829/27059","state":["exists","up"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":27049},{"type":"v1","addr":"172.21.15.157:6817","nonce":27049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":27049},{"type":"v1","addr":"172.21.15.157:6819","nonce":27049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":27049},{"type":"v1","addr":"172.21.15.157:6824","nonce":27049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":27049},{"type":"v1","addr":"172.21.15.157:6821","nonce":27049}]},"public_addr":"172.21.15.157:6817/27049","cluster_addr":"172.21.15.157:6819/27049","heartbeat_back_addr":"172.21.15.157:6824/27049","heartbeat_front_addr":"172.21.15.157:6821/27049","state":["exists","up"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":27047},{"type":"v1","addr":"172.21.15.157:6809","nonce":27047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":27047},{"type":"v1","addr":"172.21.15.157:6811","nonce":27047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":27047},{"type":"v1","addr":"172.21.15.157:6815","nonce":27047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":27047},{"type":"v1","addr":"172.21.15.157:6813","nonce":27047}]},"public_addr":"172.21.15.157:6809/27047","cluster_addr":"172.21.15.157:6811/27047","heartbeat_back_addr":"172.21.15.157:6815/27047","heartbeat_front_addr":"172.21.15.157:6813/27047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:55.154980+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:01.206226+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:36:47.401889+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:11.396380+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:35.216655+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:50.598288+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:06.008222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:07.137400+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:38:10.374 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:38:10.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:38:10.661 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:10.665 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:38:10.679 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2023-09-30T00:38:08.699374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6371516,"kb_used_data":743223624,"kb_used_omap":590,"kb_used_meta":0,"kb_avail":743223524,"statfs":{"total":767585320960,"available":761060888576,"internally_reserved":102400,"allocated":761060990976,"data_stored":761060990976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":604463,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.311088"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:37:55.361724+0000","last_change":"2023-09-30T00:37:00.203398+0000","last_active":"2023-09-30T00:37:55.361724+0000","last_peered":"2023-09-30T00:37:55.361724+0000","last_clean":"2023-09-30T00:37:55.361724+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:37:55.361724+0000","last_undegraded":"2023-09-30T00:37:55.361724+0000","last_fullsized":"2023-09-30T00:37:55.361724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_clean_scrub_stamp":"2023-09-30T00:37:00.203204+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:21.467829+0000","scrub_duration":0.20409011799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:37:55.826469+0000","last_change":"2023-09-30T00:37:15.055387+0000","last_active":"2023-09-30T00:37:55.826469+0000","last_peered":"2023-09-30T00:37:55.826469+0000","last_clean":"2023-09-30T00:37:55.826469+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:37:55.826469+0000","last_undegraded":"2023-09-30T00:37:55.826469+0000","last_fullsized":"2023-09-30T00:37:55.826469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:15.055188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:15.055188+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:24.242694+0000","scrub_duration":0.104113765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:37:55.827119+0000","last_change":"2023-09-30T00:36:59.845156+0000","last_active":"2023-09-30T00:37:55.827119+0000","last_peered":"2023-09-30T00:37:55.827119+0000","last_clean":"2023-09-30T00:37:55.827119+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:37:55.827119+0000","last_undegraded":"2023-09-30T00:37:55.827119+0000","last_fullsized":"2023-09-30T00:37:55.827119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:59.844873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:49.776230+0000","last_clean_scrub_stamp":"2023-09-30T00:36:59.844873+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:13.536638+0000","scrub_duration":0.10427676600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:37:55.826475+0000","last_change":"2023-09-30T00:37:07.344322+0000","last_active":"2023-09-30T00:37:55.826475+0000","last_peered":"2023-09-30T00:37:55.826475+0000","last_clean":"2023-09-30T00:37:55.826475+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:37:55.826475+0000","last_undegraded":"2023-09-30T00:37:55.826475+0000","last_fullsized":"2023-09-30T00:37:55.826475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:07.344143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:07.344143+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:33.080351+0000","scrub_duration":0.40480254500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.106798+0000","last_change":"2023-09-30T00:37:11.469764+0000","last_active":"2023-09-30T00:38:07.106798+0000","last_peered":"2023-09-30T00:38:07.106798+0000","last_clean":"2023-09-30T00:38:07.106798+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:38:07.106798+0000","last_undegraded":"2023-09-30T00:38:07.106798+0000","last_fullsized":"2023-09-30T00:38:07.106798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:11.469587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:11.469587+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:20.615198+0000","scrub_duration":0.204916511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:37:55.826145+0000","last_change":"2023-09-30T00:37:12.329649+0000","last_active":"2023-09-30T00:37:55.826145+0000","last_peered":"2023-09-30T00:37:55.826145+0000","last_clean":"2023-09-30T00:37:55.826145+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:37:55.826145+0000","last_undegraded":"2023-09-30T00:37:55.826145+0000","last_fullsized":"2023-09-30T00:37:55.826145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:12.329539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:34:38.283938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:12.329539+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:35.247630+0000","scrub_duration":0.002197844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-09-30T00:37:55.827061+0000","last_change":"2023-09-30T00:36:43.857145+0000","last_active":"2023-09-30T00:37:55.827061+0000","last_peered":"2023-09-30T00:37:55.827061+0000","last_clean":"2023-09-30T00:37:55.827061+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:37:55.827061+0000","last_undegraded":"2023-09-30T00:37:55.827061+0000","last_fullsized":"2023-09-30T00:37:55.827061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:43.856989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:38.924322+0000","last_clean_scrub_stamp":"2023-09-30T00:36:43.856989+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:07.193581+0000","scrub_duration":0.20415779100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.106856+0000","last_change":"2023-09-30T00:36:49.246678+0000","last_active":"2023-09-30T00:38:07.106856+0000","last_peered":"2023-09-30T00:38:07.106856+0000","last_clean":"2023-09-30T00:38:07.106856+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:38:07.106856+0000","last_undegraded":"2023-09-30T00:38:07.106856+0000","last_fullsized":"2023-09-30T00:38:07.106856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:49.246500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:36:49.246500+0000","last_clean_scrub_stamp":"2023-09-30T00:36:49.246500+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:11.878789+0000","scrub_duration":0.103874328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":230,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.086045+0000","last_change":"2023-09-30T00:38:02.085975+0000","last_active":"2023-09-30T00:38:02.086045+0000","last_peered":"2023-09-30T00:38:02.086045+0000","last_clean":"2023-09-30T00:38:02.086045+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:38:02.086045+0000","last_undegraded":"2023-09-30T00:38:02.086045+0000","last_fullsized":"2023-09-30T00:38:02.086045+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-09-30T00:38:02.085822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_clean_scrub_stamp":"2023-09-30T00:38:02.085822+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:17.769551+0000","scrub_duration":0.20558635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796864,"kb_used_data":92902532,"kb_used_omap":139,"kb_used_meta":0,"kb_avail":92902516,"statfs":{"total":95948165120,"available":95132176384,"internally_reserved":16384,"allocated":95132192768,"data_stored":95132192768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":143268,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.89900000000000002,"15min":0.86699999999999999},"min":{"1min":0.73899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2170000000000001,"5min":1.26,"15min":1.26},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.2210000000000001,"15min":1.5669999999999999},"min":{"1min":0.54500000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.153,"5min":18.120000000000001,"15min":18.120000000000001},"last":0.77000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91300000000000003,"15min":0.93000000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1719999999999999,"5min":1.224,"15min":1.224},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.71599999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1479999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.86399999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.86799999999999999,"15min":0.84999999999999998},"min":{"1min":0.66900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.97299999999999998,"15min":0.98299999999999998},"min":{"1min":0.74199999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.163,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.0549999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.96099999999999997,"15min":0.97899999999999998},"min":{"1min":0.55600000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.999,"5min":1.0209999999999999,"15min":1.044},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2889999999999999,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":1.083}]},{"osd":4,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89200000000000002,"15min":0.89900000000000002},"min":{"1min":0.64600000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0489999999999999,"5min":1.357,"15min":1.357},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.032,"15min":1.046},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3120000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.97799999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.0149999999999999,"15min":1.028},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2210000000000001,"5min":1.327,"15min":1.327},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.161,"15min":1.1779999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3220000000000001,"5min":1.393,"15min":1.393},"last":1.1080000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.091,"15min":1.1200000000000001},"min":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.244,"5min":1.45,"15min":1.45},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.143,"15min":1.169},"min":{"1min":0.79600000000000004,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.2549999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.0149999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796280,"kb_used_data":92903108,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92903100,"statfs":{"total":95948165120,"available":95132774400,"internally_reserved":8192,"allocated":95132782592,"data_stored":95132782592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57766,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.89900000000000002,"15min":0.89300000000000002},"min":{"1min":0.63200000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2849999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.93100000000000005,"15min":0.99099999999999999},"min":{"1min":0.64700000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.3140000000000001,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.997}]},{"osd":1,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.66500000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.21,"5min":1.351,"15min":1.351},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.75800000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.1539999999999999,"5min":1.341,"15min":1.341},"last":0.626}]},{"osd":2,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95799999999999996,"15min":0.96599999999999997},"min":{"1min":0.72799999999999998,"5min":0.621,"15min":0.621},"max":{"1min":1.1910000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.26},{"interface":"front","average":{"1min":1,"5min":1.02,"15min":1.0640000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3100000000000001,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.86299999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94199999999999995,"15min":0.97099999999999997},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.1559999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.075,"15min":1.1319999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.258,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.95199999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0129999999999999,"15min":1.095},"min":{"1min":0.63600000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.321,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.103,"15min":1.171},"min":{"1min":0.72199999999999998,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.335,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.137}]},{"osd":5,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0780000000000001,"15min":1.1459999999999999},"min":{"1min":0.85999999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.248,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.165},{"interface":"front","average":{"1min":1.157,"5min":1.196,"15min":1.25},"min":{"1min":0.92900000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3280000000000001,"5min":1.627,"15min":1.627},"last":1.1919999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.194,"15min":1.25},"min":{"1min":0.995,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.3740000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":1.032},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.54,"15min":1.9239999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.3340000000000001,"5min":16.701000000000001,"15min":16.701000000000001},"last":1.1080000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796352,"kb_used_data":92903044,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":92903028,"statfs":{"total":95948165120,"available":95132700672,"internally_reserved":16384,"allocated":95132717056,"data_stored":95132717056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53153,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91900000000000004,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2390000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.89700000000000002,"15min":0.878},"min":{"1min":0.55200000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.748}]},{"osd":1,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":0.871},"min":{"1min":0.63900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.405,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93600000000000005,"15min":0.91000000000000003},"min":{"1min":0.72999999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.80600000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.89600000000000002},"min":{"1min":0.59399999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98999999999999999,"15min":0.96699999999999997},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.351,"5min":1.546,"15min":1.546},"last":1.099}]},{"osd":3,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.69699999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":1.012},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":1.0249999999999999},"min":{"1min":0.76600000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3799999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.0549999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93700000000000006,"15min":0.96299999999999997},"min":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.24,"5min":1.425,"15min":1.425},"last":1.246},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.2110000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0589999999999999,"15min":1.0840000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.90200000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4750000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.3149999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4730000000000001,"5min":1.514,"15min":1.514},"last":1.341},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3759999999999999},"min":{"1min":0.93300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6519999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":1.379}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796216,"kb_used_data":92903180,"kb_used_omap":39,"kb_used_meta":0,"kb_avail":92903164,"statfs":{"total":95948165120,"available":95132839936,"internally_reserved":16384,"allocated":95132856320,"data_stored":95132856320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40108,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.93700000000000006,"15min":0.98299999999999998},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2410000000000001,"5min":1.446,"15min":1.446},"last":1.077},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88700000000000001,"15min":0.92500000000000004},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.103,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.111}]},{"osd":1,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.623,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.141},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.034,"15min":1.1419999999999999},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5760000000000001,"5min":5.1319999999999997,"15min":5.1319999999999997},"last":1.411}]},{"osd":2,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.92600000000000005,"15min":0.94399999999999995},"min":{"1min":0.65800000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.1890000000000001,"5min":1.456,"15min":1.456},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.008,"15min":1.054},"min":{"1min":0.76900000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.224,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1919999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.995,"15min":1.0529999999999999},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2130000000000001,"5min":2.048,"15min":2.048},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.03,"15min":1.0620000000000001},"min":{"1min":0.73099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.218,"5min":1.571,"15min":1.571},"last":1.3029999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.95799999999999996,"15min":1.01},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1910000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.091,"15min":1.0960000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.405,"5min":1.492,"15min":1.492},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1479999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2969999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.236},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.163,"15min":1.224},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.3799999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.93600000000000005}]},{"osd":7,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.159,"15min":1.23},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.323,"5min":2.851,"15min":2.851},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.24,"15min":1.3049999999999999},"min":{"1min":0.96699999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5029999999999999,"5min":1.887,"15min":1.887},"last":1.345}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796292,"kb_used_data":92903096,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903088,"statfs":{"total":95948165120,"available":95132762112,"internally_reserved":8192,"allocated":95132770304,"data_stored":95132770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48990,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66900000000000004,"15min":0.66100000000000003},"min":{"1min":0.49299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1890000000000001,"5min":1.345,"15min":1.345},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76900000000000002,"15min":0.72699999999999998},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0269999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.94499999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.54400000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.016,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.1459999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.81200000000000006,"15min":0.80000000000000004},"min":{"1min":0.57699999999999996,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.099,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.55700000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70999999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.76200000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.373,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0109999999999999,"15min":0.97699999999999998},"min":{"1min":0.77400000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.343,"5min":1.466,"15min":1.466},"last":1.004}]},{"osd":5,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0489999999999999,"15min":1.0369999999999999},"min":{"1min":0.77100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.538,"5min":2.169,"15min":2.169},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0840000000000001,"15min":1.0600000000000001},"min":{"1min":0.94999999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.375,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.109}]},{"osd":6,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.099,"15min":1.095},"min":{"1min":0.83599999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.472,"5min":1.474,"15min":1.474},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.1459999999999999,"15min":1.1259999999999999},"min":{"1min":1.052,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.46,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.0369999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.171,"15min":1.153},"min":{"1min":0.92800000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.4910000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.306},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1699999999999999,"15min":1.1659999999999999},"min":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.4259999999999999,"5min":1.554,"15min":1.554},"last":1.1870000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796276,"kb_used_data":92903112,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903104,"statfs":{"total":95948165120,"available":95132778496,"internally_reserved":8192,"allocated":95132786688,"data_stored":95132786688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53271,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.76500000000000001,"15min":0.83299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.871,"5min":4.5780000000000003,"15min":4.5780000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.85899999999999999,"15min":0.92900000000000005},"min":{"1min":0.56899999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.171,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.71999999999999997}]},{"osd":1,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79900000000000004,"15min":0.82599999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0620000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.77700000000000002,"15min":0.77000000000000002},"min":{"1min":0.55700000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.038,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.877}]},{"osd":3,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.57899999999999996,"5min":0.505,"15min":0.505},"max":{"1min":1.044,"5min":1.081,"15min":1.081},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91600000000000004},"min":{"1min":0.54000000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.095,"5min":1.49,"15min":1.49},"last":0.77800000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.008,"15min":1.012},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1990000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.998,"5min":1.0189999999999999,"15min":1.0169999999999999},"min":{"1min":0.82599999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2609999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.03}]},{"osd":5,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0089999999999999,"15min":1.004},"min":{"1min":0.84599999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2490000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.121,"15min":1.131},"min":{"1min":0.83499999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.3260000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.0649999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1040000000000001,"15min":1.1279999999999999},"min":{"1min":0.86799999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.369,"15min":1.1719999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.425,"5min":29.962,"15min":29.962},"last":1.1240000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.157,"15min":1.1830000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.4259999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.159},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.167,"15min":1.171},"min":{"1min":0.95399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.99099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796288,"kb_used_data":92903104,"kb_used_omap":73,"kb_used_meta":0,"kb_avail":92903092,"statfs":{"total":95948165120,"available":95132766208,"internally_reserved":12288,"allocated":95132778496,"data_stored":95132778496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75671,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68300000000000005,"15min":0.69599999999999995},"min":{"1min":0.5,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.0309999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.498},{"interface":"front","average":{"1min":0.746,"5min":0.76100000000000001,"15min":0.79300000000000004},"min":{"1min":0.56999999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.0900000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.77500000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.70199999999999996,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81200000000000006,"15min":0.81699999999999995},"min":{"1min":0.55900000000000005,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.399,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.995}]},{"osd":3,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.48899999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2569999999999999,"5min":1.282,"15min":1.282},"last":0.871},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.44600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1499999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.66400000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.0169999999999999,"15min":1.0429999999999999},"min":{"1min":0.752,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.2669999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.997,"15min":1.0049999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2709999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.83999999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.018,"15min":1.018},"min":{"1min":0.74299999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.288,"5min":1.593,"15min":1.593},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1259999999999999,"15min":1.1459999999999999},"min":{"1min":0.85299999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.3540000000000001,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.0800000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.135,"15min":1.1639999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.476,"5min":1.962,"15min":1.962},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.113,"15min":1.115},"min":{"1min":0.78600000000000003,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4670000000000001,"5min":1.665,"15min":1.665},"last":0.92500000000000004}]},{"osd":7,"last update":"Sat Sep 30 00:37:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.175,"15min":1.222},"min":{"1min":0.79700000000000004,"5min":0.749,"15min":0.749},"max":{"1min":1.46,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":1.151},{"interface":"front","average":{"1min":1.179,"5min":3.1749999999999998,"15min":5.1790000000000003},"min":{"1min":0.83299999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.393,"5min":129.49100000000001,"15min":129.49100000000001},"last":1.171}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796948,"kb_used_data":92902448,"kb_used_omap":129,"kb_used_meta":0,"kb_avail":92902432,"statfs":{"total":95948165120,"available":95132090368,"internally_reserved":16384,"allocated":95132106752,"data_stored":95132106752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":132236,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69399999999999995,"15min":0.72399999999999998},"min":{"1min":0.51200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.96499999999999997,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.745,"15min":0.70999999999999996},"min":{"1min":0.61499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.82599999999999996}]},{"osd":2,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.746,"15min":0.752},"min":{"1min":0.59199999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.028,"5min":1.119,"15min":1.119},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.61699999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.75800000000000001,"15min":0.73499999999999999},"min":{"1min":0.56000000000000005,"5min":0.373,"15min":0.373},"max":{"1min":1.085,"5min":1.292,"15min":1.292},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88600000000000001,"15min":0.86599999999999999},"min":{"1min":0.57999999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2669999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.98199999999999998,"15min":0.97599999999999998},"min":{"1min":0.81999999999999995,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.395,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.032,"15min":1.0980000000000001},"min":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1299999999999999,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.996,"15min":1.002},"min":{"1min":0.76600000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.26,"5min":2.165,"15min":2.165},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.151,"5min":1.1419999999999999,"15min":1.1439999999999999},"min":{"1min":0.78000000000000003,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.498,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.218}]},{"osd":6,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1399999999999999,"15min":1.115},"min":{"1min":0.83299999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.601,"5min":1.615,"15min":1.615},"last":1.167},{"interface":"front","average":{"1min":1.149,"5min":1.119,"15min":1.095},"min":{"1min":0.92100000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4410000000000001,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.0389999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1399999999999999,"15min":1.1299999999999999},"min":{"1min":0.80600000000000005,"5min":0.755,"15min":0.755},"max":{"1min":1.5409999999999999,"5min":1.71,"15min":1.71},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.129,"5min":2.7389999999999999,"15min":4.3449999999999998},"min":{"1min":0.94499999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.389,"5min":80.903000000000006,"15min":80.903000000000006},"last":1.0669999999999999}]}]}],"pool_statfs":[]}} 2023-09-30T00:38:10.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:38:10.967 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:38:10.982 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2023-09-30T00:38:10.699955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6371656,"kb_used_data":743223496,"kb_used_omap":590,"kb_used_meta":0,"kb_avail":743223384,"statfs":{"total":767585320960,"available":761060745216,"internally_reserved":114688,"allocated":761060859904,"data_stored":761060859904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":604891,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002226"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.067656+0000","last_change":"2023-09-30T00:37:00.203398+0000","last_active":"2023-09-30T00:38:02.067656+0000","last_peered":"2023-09-30T00:38:02.067656+0000","last_clean":"2023-09-30T00:38:02.067656+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:38:02.067656+0000","last_undegraded":"2023-09-30T00:38:02.067656+0000","last_fullsized":"2023-09-30T00:38:02.067656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_clean_scrub_stamp":"2023-09-30T00:37:00.203204+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:21.467829+0000","scrub_duration":0.20409011799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:06.512432+0000","last_change":"2023-09-30T00:37:15.055387+0000","last_active":"2023-09-30T00:38:06.512432+0000","last_peered":"2023-09-30T00:38:06.512432+0000","last_clean":"2023-09-30T00:38:06.512432+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:38:06.512432+0000","last_undegraded":"2023-09-30T00:38:06.512432+0000","last_fullsized":"2023-09-30T00:38:06.512432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:15.055188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:15.055188+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:24.242694+0000","scrub_duration":0.104113765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:01.390516+0000","last_change":"2023-09-30T00:36:59.845156+0000","last_active":"2023-09-30T00:38:01.390516+0000","last_peered":"2023-09-30T00:38:01.390516+0000","last_clean":"2023-09-30T00:38:01.390516+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:38:01.390516+0000","last_undegraded":"2023-09-30T00:38:01.390516+0000","last_fullsized":"2023-09-30T00:38:01.390516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:59.844873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:49.776230+0000","last_clean_scrub_stamp":"2023-09-30T00:36:59.844873+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:13.536638+0000","scrub_duration":0.10427676600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:06.512417+0000","last_change":"2023-09-30T00:37:07.344322+0000","last_active":"2023-09-30T00:38:06.512417+0000","last_peered":"2023-09-30T00:38:06.512417+0000","last_clean":"2023-09-30T00:38:06.512417+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:38:06.512417+0000","last_undegraded":"2023-09-30T00:38:06.512417+0000","last_fullsized":"2023-09-30T00:38:06.512417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:07.344143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:07.344143+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:33.080351+0000","scrub_duration":0.40480254500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.106798+0000","last_change":"2023-09-30T00:37:11.469764+0000","last_active":"2023-09-30T00:38:07.106798+0000","last_peered":"2023-09-30T00:38:07.106798+0000","last_clean":"2023-09-30T00:38:07.106798+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:38:07.106798+0000","last_undegraded":"2023-09-30T00:38:07.106798+0000","last_fullsized":"2023-09-30T00:38:07.106798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:11.469587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:11.469587+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:20.615198+0000","scrub_duration":0.204916511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.636280+0000","last_change":"2023-09-30T00:37:12.329649+0000","last_active":"2023-09-30T00:38:02.636280+0000","last_peered":"2023-09-30T00:38:02.636280+0000","last_clean":"2023-09-30T00:38:02.636280+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:38:02.636280+0000","last_undegraded":"2023-09-30T00:38:02.636280+0000","last_fullsized":"2023-09-30T00:38:02.636280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:12.329539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:34:38.283938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:12.329539+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:35.247630+0000","scrub_duration":0.002197844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:01.390389+0000","last_change":"2023-09-30T00:36:43.857145+0000","last_active":"2023-09-30T00:38:01.390389+0000","last_peered":"2023-09-30T00:38:01.390389+0000","last_clean":"2023-09-30T00:38:01.390389+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:38:01.390389+0000","last_undegraded":"2023-09-30T00:38:01.390389+0000","last_fullsized":"2023-09-30T00:38:01.390389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:43.856989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:38.924322+0000","last_clean_scrub_stamp":"2023-09-30T00:36:43.856989+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:07.193581+0000","scrub_duration":0.20415779100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.106856+0000","last_change":"2023-09-30T00:36:49.246678+0000","last_active":"2023-09-30T00:38:07.106856+0000","last_peered":"2023-09-30T00:38:07.106856+0000","last_clean":"2023-09-30T00:38:07.106856+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:38:07.106856+0000","last_undegraded":"2023-09-30T00:38:07.106856+0000","last_fullsized":"2023-09-30T00:38:07.106856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:49.246500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:36:49.246500+0000","last_clean_scrub_stamp":"2023-09-30T00:36:49.246500+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:11.878789+0000","scrub_duration":0.103874328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":231,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.086089+0000","last_change":"2023-09-30T00:38:02.085975+0000","last_active":"2023-09-30T00:38:02.086089+0000","last_peered":"2023-09-30T00:38:02.086089+0000","last_clean":"2023-09-30T00:38:02.086089+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:38:02.086089+0000","last_undegraded":"2023-09-30T00:38:02.086089+0000","last_fullsized":"2023-09-30T00:38:02.086089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-09-30T00:38:02.085822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_clean_scrub_stamp":"2023-09-30T00:38:02.085822+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:17.769551+0000","scrub_duration":0.20558635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796872,"kb_used_data":92902520,"kb_used_omap":139,"kb_used_meta":0,"kb_avail":92902508,"statfs":{"total":95948165120,"available":95132168192,"internally_reserved":12288,"allocated":95132180480,"data_stored":95132180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":143356,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.89900000000000002,"15min":0.86699999999999999},"min":{"1min":0.73899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2170000000000001,"5min":1.26,"15min":1.26},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.2210000000000001,"15min":1.5669999999999999},"min":{"1min":0.54500000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.153,"5min":18.120000000000001,"15min":18.120000000000001},"last":1.147}]},{"osd":1,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91300000000000003,"15min":0.93000000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1719999999999999,"5min":1.224,"15min":1.224},"last":0.87},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.71599999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1479999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.306}]},{"osd":2,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.86799999999999999,"15min":0.84999999999999998},"min":{"1min":0.66900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.97299999999999998,"15min":0.98299999999999998},"min":{"1min":0.74199999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.163,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.2190000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.96099999999999997,"15min":0.97899999999999998},"min":{"1min":0.55600000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.0209999999999999,"15min":1.044},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2889999999999999,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":1.3959999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89200000000000002,"15min":0.89900000000000002},"min":{"1min":0.64600000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0489999999999999,"5min":1.357,"15min":1.357},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.032,"15min":1.046},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3120000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.93200000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.0149999999999999,"15min":1.028},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2210000000000001,"5min":1.327,"15min":1.327},"last":1.252},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.161,"15min":1.1779999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3220000000000001,"5min":1.393,"15min":1.393},"last":1.1180000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.091,"15min":1.1200000000000001},"min":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.244,"5min":1.45,"15min":1.45},"last":1.05},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.143,"15min":1.169},"min":{"1min":0.79600000000000004,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.2549999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.351}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796296,"kb_used_data":92903096,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92903084,"statfs":{"total":95948165120,"available":95132758016,"internally_reserved":12288,"allocated":95132770304,"data_stored":95132770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57854,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.90800000000000003,"15min":0.89600000000000002},"min":{"1min":0.65800000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2709999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91200000000000003,"15min":0.98399999999999999},"min":{"1min":0.71699999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.83199999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.64500000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.23,"5min":1.351,"15min":1.351},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.95599999999999996,"15min":0.96799999999999997},"min":{"1min":0.626,"5min":0.621,"15min":0.621},"max":{"1min":1.2490000000000001,"5min":1.341,"15min":1.341},"last":1.05}]},{"osd":2,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.96599999999999997},"min":{"1min":0.69199999999999995,"5min":0.621,"15min":0.621},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.995,"5min":1.002,"15min":1.0580000000000001},"min":{"1min":0.68999999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.304,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.745,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.012,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0580000000000001,"15min":1.1259999999999999},"min":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3180000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.2}]},{"osd":4,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.97299999999999998,"15min":1.081},"min":{"1min":0.68500000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.248,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.248},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0840000000000001,"15min":1.1639999999999999},"min":{"1min":0.92200000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.325,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.081}]},{"osd":5,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0509999999999999,"15min":1.137},"min":{"1min":0.68300000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1950000000000001,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.169,"15min":1.2410000000000001},"min":{"1min":0.753,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3600000000000001,"5min":1.627,"15min":1.627},"last":1.296}]},{"osd":7,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.173,"15min":1.2430000000000001},"min":{"1min":0.90500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.3540000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.333,"15min":1.855},"min":{"1min":0.89600000000000002,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.3620000000000001,"5min":16.701000000000001,"15min":16.701000000000001},"last":1.016}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796352,"kb_used_data":92903044,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":92903028,"statfs":{"total":95948165120,"available":95132700672,"internally_reserved":16384,"allocated":95132717056,"data_stored":95132717056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53153,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91900000000000004,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2390000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.89700000000000002,"15min":0.878},"min":{"1min":0.55200000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.748}]},{"osd":1,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":0.871},"min":{"1min":0.63900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.405,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93600000000000005,"15min":0.91000000000000003},"min":{"1min":0.72999999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.80600000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.89600000000000002},"min":{"1min":0.59399999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98999999999999999,"15min":0.96699999999999997},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.351,"5min":1.546,"15min":1.546},"last":1.099}]},{"osd":3,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.69699999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":1.012},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":1.0249999999999999},"min":{"1min":0.76600000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3799999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.0549999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93700000000000006,"15min":0.96299999999999997},"min":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.24,"5min":1.425,"15min":1.425},"last":1.246},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.2110000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0589999999999999,"15min":1.0840000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.90200000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4750000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.3149999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4730000000000001,"5min":1.514,"15min":1.514},"last":1.341},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3759999999999999},"min":{"1min":0.93300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6519999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":1.379}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796216,"kb_used_data":92903180,"kb_used_omap":39,"kb_used_meta":0,"kb_avail":92903164,"statfs":{"total":95948165120,"available":95132839936,"internally_reserved":16384,"allocated":95132856320,"data_stored":95132856320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40108,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.93700000000000006,"15min":0.98299999999999998},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2410000000000001,"5min":1.446,"15min":1.446},"last":1.077},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88700000000000001,"15min":0.92500000000000004},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.103,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.111}]},{"osd":1,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.623,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.141},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.034,"15min":1.1419999999999999},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5760000000000001,"5min":5.1319999999999997,"15min":5.1319999999999997},"last":1.411}]},{"osd":2,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.92600000000000005,"15min":0.94399999999999995},"min":{"1min":0.65800000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.1890000000000001,"5min":1.456,"15min":1.456},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.008,"15min":1.054},"min":{"1min":0.76900000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.224,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1919999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.995,"15min":1.0529999999999999},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2130000000000001,"5min":2.048,"15min":2.048},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.03,"15min":1.0620000000000001},"min":{"1min":0.73099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.218,"5min":1.571,"15min":1.571},"last":1.3029999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.95799999999999996,"15min":1.01},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1910000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.091,"15min":1.0960000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.405,"5min":1.492,"15min":1.492},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1479999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2969999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.236},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.163,"15min":1.224},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.3799999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.93600000000000005}]},{"osd":7,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.159,"15min":1.23},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.323,"5min":2.851,"15min":2.851},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.24,"15min":1.3049999999999999},"min":{"1min":0.96699999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5029999999999999,"5min":1.887,"15min":1.887},"last":1.345}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796292,"kb_used_data":92903096,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903088,"statfs":{"total":95948165120,"available":95132762112,"internally_reserved":8192,"allocated":95132770304,"data_stored":95132770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48990,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66900000000000004,"15min":0.66100000000000003},"min":{"1min":0.49299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1890000000000001,"5min":1.345,"15min":1.345},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76900000000000002,"15min":0.72699999999999998},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0269999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.94499999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.54400000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.016,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.1459999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.81200000000000006,"15min":0.80000000000000004},"min":{"1min":0.57699999999999996,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.099,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.55700000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70999999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.76200000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.373,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0109999999999999,"15min":0.97699999999999998},"min":{"1min":0.77400000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.343,"5min":1.466,"15min":1.466},"last":1.004}]},{"osd":5,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0489999999999999,"15min":1.0369999999999999},"min":{"1min":0.77100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.538,"5min":2.169,"15min":2.169},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0840000000000001,"15min":1.0600000000000001},"min":{"1min":0.94999999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.375,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.109}]},{"osd":6,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.099,"15min":1.095},"min":{"1min":0.83599999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.472,"5min":1.474,"15min":1.474},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.1459999999999999,"15min":1.1259999999999999},"min":{"1min":1.052,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.46,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.0369999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.171,"15min":1.153},"min":{"1min":0.92800000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.4910000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.306},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1699999999999999,"15min":1.1659999999999999},"min":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.4259999999999999,"5min":1.554,"15min":1.554},"last":1.1870000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796348,"kb_used_data":92903048,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903032,"statfs":{"total":95948165120,"available":95132704768,"internally_reserved":16384,"allocated":95132721152,"data_stored":95132721152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53397,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72699999999999998,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":4.5780000000000003,"15min":4.5780000000000003},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82299999999999995,"15min":0.91700000000000004},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0449999999999999,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.89800000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.77500000000000002,"15min":0.81799999999999995},"min":{"1min":0.51600000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.0369999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.495,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.071,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.62}]},{"osd":3,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.878,"15min":0.91200000000000003},"min":{"1min":0.64600000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.177,"5min":1.49,"15min":1.49},"last":0.70099999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.998,"15min":1.0089999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.216,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0089999999999999,"15min":1.014},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1839999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.153}]},{"osd":5,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.008,"15min":1.004},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3140000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.454},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3859999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.105}]},{"osd":6,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.085,"15min":1.1220000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5569999999999999,"5min":1.607,"15min":1.607},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.3680000000000001,"15min":1.1719999999999999},"min":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3440000000000001,"5min":29.962,"15min":29.962},"last":1.337}]},{"osd":7,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1419999999999999,"15min":1.1779999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.6160000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1679999999999999},"min":{"1min":0.88,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.2050000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796332,"kb_used_data":92903064,"kb_used_omap":74,"kb_used_meta":0,"kb_avail":92903048,"statfs":{"total":95948165120,"available":95132721152,"internally_reserved":16384,"allocated":95132737536,"data_stored":95132737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75797,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.67500000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.98699999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73999999999999999,"15min":0.78600000000000003},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.0629999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.0629999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70299999999999996,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.90700000000000003,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.879},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3839999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.83399999999999996}]},{"osd":3,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85099999999999998,"15min":0.88600000000000001},"min":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2,"5min":1.282,"15min":1.282},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84299999999999997,"15min":0.85999999999999999},"min":{"1min":0.65700000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1120000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0049999999999999,"15min":1.0389999999999999},"min":{"1min":0.81699999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3100000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.018,"5min":0.999,"15min":1.006},"min":{"1min":0.73799999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3129999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.02}]},{"osd":5,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.014,"15min":1.0169999999999999},"min":{"1min":0.72999999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.1919999999999999,"5min":1.593,"15min":1.593},"last":1.157},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.88300000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.581,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.98499999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1279999999999999,"15min":1.1619999999999999},"min":{"1min":0.92800000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.4870000000000001,"5min":1.962,"15min":1.962},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.115,"15min":1.115},"min":{"1min":0.83599999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.3380000000000001,"5min":1.665,"15min":1.665},"last":1.224}]},{"osd":7,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1639999999999999,"15min":1.218},"min":{"1min":0.98199999999999998,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":1.111},{"interface":"front","average":{"1min":1.169,"5min":2.173,"15min":4.8449999999999998},"min":{"1min":0.88400000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.542,"5min":129.49100000000001,"15min":129.49100000000001},"last":1.3859999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796948,"kb_used_data":92902448,"kb_used_omap":129,"kb_used_meta":0,"kb_avail":92902432,"statfs":{"total":95948165120,"available":95132090368,"internally_reserved":16384,"allocated":95132106752,"data_stored":95132106752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":132236,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69399999999999995,"15min":0.72399999999999998},"min":{"1min":0.51200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.96499999999999997,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.745,"15min":0.70999999999999996},"min":{"1min":0.61499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.82599999999999996}]},{"osd":2,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.746,"15min":0.752},"min":{"1min":0.59199999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.028,"5min":1.119,"15min":1.119},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.61699999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.75800000000000001,"15min":0.73499999999999999},"min":{"1min":0.56000000000000005,"5min":0.373,"15min":0.373},"max":{"1min":1.085,"5min":1.292,"15min":1.292},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88600000000000001,"15min":0.86599999999999999},"min":{"1min":0.57999999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2669999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.98199999999999998,"15min":0.97599999999999998},"min":{"1min":0.81999999999999995,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.395,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.032,"15min":1.0980000000000001},"min":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1299999999999999,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.996,"15min":1.002},"min":{"1min":0.76600000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.26,"5min":2.165,"15min":2.165},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.151,"5min":1.1419999999999999,"15min":1.1439999999999999},"min":{"1min":0.78000000000000003,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.498,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.218}]},{"osd":6,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1399999999999999,"15min":1.115},"min":{"1min":0.83299999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.601,"5min":1.615,"15min":1.615},"last":1.167},{"interface":"front","average":{"1min":1.149,"5min":1.119,"15min":1.095},"min":{"1min":0.92100000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4410000000000001,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.0389999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1399999999999999,"15min":1.1299999999999999},"min":{"1min":0.80600000000000005,"5min":0.755,"15min":0.755},"max":{"1min":1.5409999999999999,"5min":1.71,"15min":1.71},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.129,"5min":2.7389999999999999,"15min":4.3449999999999998},"min":{"1min":0.94499999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.389,"5min":80.903000000000006,"15min":80.903000000000006},"last":1.0669999999999999}]}]}],"pool_statfs":[]}} 2023-09-30T00:38:10.984 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:38:10.984 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:38:11.274 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:11.276 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:38:11.286 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2023-09-30T00:38:10.699955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6371656,"kb_used_data":743223496,"kb_used_omap":590,"kb_used_meta":0,"kb_avail":743223384,"statfs":{"total":767585320960,"available":761060745216,"internally_reserved":114688,"allocated":761060859904,"data_stored":761060859904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":604891,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002226"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.067656+0000","last_change":"2023-09-30T00:37:00.203398+0000","last_active":"2023-09-30T00:38:02.067656+0000","last_peered":"2023-09-30T00:38:02.067656+0000","last_clean":"2023-09-30T00:38:02.067656+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:38:02.067656+0000","last_undegraded":"2023-09-30T00:38:02.067656+0000","last_fullsized":"2023-09-30T00:38:02.067656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_clean_scrub_stamp":"2023-09-30T00:37:00.203204+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:21.467829+0000","scrub_duration":0.20409011799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:06.512432+0000","last_change":"2023-09-30T00:37:15.055387+0000","last_active":"2023-09-30T00:38:06.512432+0000","last_peered":"2023-09-30T00:38:06.512432+0000","last_clean":"2023-09-30T00:38:06.512432+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:38:06.512432+0000","last_undegraded":"2023-09-30T00:38:06.512432+0000","last_fullsized":"2023-09-30T00:38:06.512432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:15.055188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:15.055188+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:24.242694+0000","scrub_duration":0.104113765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:01.390516+0000","last_change":"2023-09-30T00:36:59.845156+0000","last_active":"2023-09-30T00:38:01.390516+0000","last_peered":"2023-09-30T00:38:01.390516+0000","last_clean":"2023-09-30T00:38:01.390516+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:38:01.390516+0000","last_undegraded":"2023-09-30T00:38:01.390516+0000","last_fullsized":"2023-09-30T00:38:01.390516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:59.844873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:49.776230+0000","last_clean_scrub_stamp":"2023-09-30T00:36:59.844873+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:13.536638+0000","scrub_duration":0.10427676600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:06.512417+0000","last_change":"2023-09-30T00:37:07.344322+0000","last_active":"2023-09-30T00:38:06.512417+0000","last_peered":"2023-09-30T00:38:06.512417+0000","last_clean":"2023-09-30T00:38:06.512417+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:38:06.512417+0000","last_undegraded":"2023-09-30T00:38:06.512417+0000","last_fullsized":"2023-09-30T00:38:06.512417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:07.344143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:07.344143+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:33.080351+0000","scrub_duration":0.40480254500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.106798+0000","last_change":"2023-09-30T00:37:11.469764+0000","last_active":"2023-09-30T00:38:07.106798+0000","last_peered":"2023-09-30T00:38:07.106798+0000","last_clean":"2023-09-30T00:38:07.106798+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:38:07.106798+0000","last_undegraded":"2023-09-30T00:38:07.106798+0000","last_fullsized":"2023-09-30T00:38:07.106798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:11.469587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:11.469587+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:20.615198+0000","scrub_duration":0.204916511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.636280+0000","last_change":"2023-09-30T00:37:12.329649+0000","last_active":"2023-09-30T00:38:02.636280+0000","last_peered":"2023-09-30T00:38:02.636280+0000","last_clean":"2023-09-30T00:38:02.636280+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:38:02.636280+0000","last_undegraded":"2023-09-30T00:38:02.636280+0000","last_fullsized":"2023-09-30T00:38:02.636280+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:12.329539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:34:38.283938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:12.329539+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:35.247630+0000","scrub_duration":0.002197844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:01.390389+0000","last_change":"2023-09-30T00:36:43.857145+0000","last_active":"2023-09-30T00:38:01.390389+0000","last_peered":"2023-09-30T00:38:01.390389+0000","last_clean":"2023-09-30T00:38:01.390389+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:38:01.390389+0000","last_undegraded":"2023-09-30T00:38:01.390389+0000","last_fullsized":"2023-09-30T00:38:01.390389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:43.856989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:38.924322+0000","last_clean_scrub_stamp":"2023-09-30T00:36:43.856989+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:07.193581+0000","scrub_duration":0.20415779100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":69,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.106856+0000","last_change":"2023-09-30T00:36:49.246678+0000","last_active":"2023-09-30T00:38:07.106856+0000","last_peered":"2023-09-30T00:38:07.106856+0000","last_clean":"2023-09-30T00:38:07.106856+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:38:07.106856+0000","last_undegraded":"2023-09-30T00:38:07.106856+0000","last_fullsized":"2023-09-30T00:38:07.106856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:49.246500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:36:49.246500+0000","last_clean_scrub_stamp":"2023-09-30T00:36:49.246500+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:11.878789+0000","scrub_duration":0.103874328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":231,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-09-30T00:38:02.086089+0000","last_change":"2023-09-30T00:38:02.085975+0000","last_active":"2023-09-30T00:38:02.086089+0000","last_peered":"2023-09-30T00:38:02.086089+0000","last_clean":"2023-09-30T00:38:02.086089+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:38:02.086089+0000","last_undegraded":"2023-09-30T00:38:02.086089+0000","last_fullsized":"2023-09-30T00:38:02.086089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-09-30T00:38:02.085822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_clean_scrub_stamp":"2023-09-30T00:38:02.085822+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:17.769551+0000","scrub_duration":0.20558635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796872,"kb_used_data":92902520,"kb_used_omap":139,"kb_used_meta":0,"kb_avail":92902508,"statfs":{"total":95948165120,"available":95132168192,"internally_reserved":12288,"allocated":95132180480,"data_stored":95132180480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":143356,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.89900000000000002,"15min":0.86699999999999999},"min":{"1min":0.73899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2170000000000001,"5min":1.26,"15min":1.26},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.2210000000000001,"15min":1.5669999999999999},"min":{"1min":0.54500000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.153,"5min":18.120000000000001,"15min":18.120000000000001},"last":1.147}]},{"osd":1,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91300000000000003,"15min":0.93000000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1719999999999999,"5min":1.224,"15min":1.224},"last":0.87},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.92300000000000004,"15min":0.95299999999999996},"min":{"1min":0.71599999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1479999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.306}]},{"osd":2,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.86799999999999999,"15min":0.84999999999999998},"min":{"1min":0.66900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.97299999999999998,"15min":0.98299999999999998},"min":{"1min":0.74199999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.163,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.2190000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.96099999999999997,"15min":0.97899999999999998},"min":{"1min":0.55600000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.0209999999999999,"15min":1.044},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2889999999999999,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":1.3959999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89200000000000002,"15min":0.89900000000000002},"min":{"1min":0.64600000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0489999999999999,"5min":1.357,"15min":1.357},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.032,"15min":1.046},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3120000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.93200000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.0149999999999999,"15min":1.028},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.2210000000000001,"5min":1.327,"15min":1.327},"last":1.252},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.161,"15min":1.1779999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3220000000000001,"5min":1.393,"15min":1.393},"last":1.1180000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:37:06 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.091,"15min":1.1200000000000001},"min":{"1min":0.77900000000000003,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.244,"5min":1.45,"15min":1.45},"last":1.05},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.143,"15min":1.169},"min":{"1min":0.79600000000000004,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.2549999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.351}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796296,"kb_used_data":92903096,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92903084,"statfs":{"total":95948165120,"available":95132758016,"internally_reserved":12288,"allocated":95132770304,"data_stored":95132770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57854,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.90800000000000003,"15min":0.89600000000000002},"min":{"1min":0.65800000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2709999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91200000000000003,"15min":0.98399999999999999},"min":{"1min":0.71699999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.83199999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.64500000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.23,"5min":1.351,"15min":1.351},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.95599999999999996,"15min":0.96799999999999997},"min":{"1min":0.626,"5min":0.621,"15min":0.621},"max":{"1min":1.2490000000000001,"5min":1.341,"15min":1.341},"last":1.05}]},{"osd":2,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.96599999999999997},"min":{"1min":0.69199999999999995,"5min":0.621,"15min":0.621},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.995,"5min":1.002,"15min":1.0580000000000001},"min":{"1min":0.68999999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.304,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.745,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.012,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0580000000000001,"15min":1.1259999999999999},"min":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3180000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.2}]},{"osd":4,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.97299999999999998,"15min":1.081},"min":{"1min":0.68500000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.248,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.248},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0840000000000001,"15min":1.1639999999999999},"min":{"1min":0.92200000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.325,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.081}]},{"osd":5,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0509999999999999,"15min":1.137},"min":{"1min":0.68300000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1950000000000001,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.169,"15min":1.2410000000000001},"min":{"1min":0.753,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3600000000000001,"5min":1.627,"15min":1.627},"last":1.296}]},{"osd":7,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.173,"15min":1.2430000000000001},"min":{"1min":0.90500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.3540000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.333,"15min":1.855},"min":{"1min":0.89600000000000002,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.3620000000000001,"5min":16.701000000000001,"15min":16.701000000000001},"last":1.016}]}]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796352,"kb_used_data":92903044,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":92903028,"statfs":{"total":95948165120,"available":95132700672,"internally_reserved":16384,"allocated":95132717056,"data_stored":95132717056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53153,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91900000000000004,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2390000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.89700000000000002,"15min":0.878},"min":{"1min":0.55200000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.748}]},{"osd":1,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":0.871},"min":{"1min":0.63900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.405,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93600000000000005,"15min":0.91000000000000003},"min":{"1min":0.72999999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.80600000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.89600000000000002},"min":{"1min":0.59399999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98999999999999999,"15min":0.96699999999999997},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.351,"5min":1.546,"15min":1.546},"last":1.099}]},{"osd":3,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.69699999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":1.012},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":1.0249999999999999},"min":{"1min":0.76600000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3799999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.0549999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93700000000000006,"15min":0.96299999999999997},"min":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.24,"5min":1.425,"15min":1.425},"last":1.246},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.2110000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0589999999999999,"15min":1.0840000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.90200000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4750000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.3149999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4730000000000001,"5min":1.514,"15min":1.514},"last":1.341},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3759999999999999},"min":{"1min":0.93300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6519999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":1.379}]}]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796216,"kb_used_data":92903180,"kb_used_omap":39,"kb_used_meta":0,"kb_avail":92903164,"statfs":{"total":95948165120,"available":95132839936,"internally_reserved":16384,"allocated":95132856320,"data_stored":95132856320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40108,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.93700000000000006,"15min":0.98299999999999998},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2410000000000001,"5min":1.446,"15min":1.446},"last":1.077},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88700000000000001,"15min":0.92500000000000004},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.103,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.111}]},{"osd":1,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.621,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.623,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.141},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.034,"15min":1.1419999999999999},"min":{"1min":0.67500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5760000000000001,"5min":5.1319999999999997,"15min":5.1319999999999997},"last":1.411}]},{"osd":2,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.92600000000000005,"15min":0.94399999999999995},"min":{"1min":0.65800000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.1890000000000001,"5min":1.456,"15min":1.456},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.008,"15min":1.054},"min":{"1min":0.76900000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.224,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.1919999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.995,"15min":1.0529999999999999},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2130000000000001,"5min":2.048,"15min":2.048},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.03,"15min":1.0620000000000001},"min":{"1min":0.73099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.218,"5min":1.571,"15min":1.571},"last":1.3029999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.95799999999999996,"15min":1.01},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1910000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.091,"15min":1.0960000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.405,"5min":1.492,"15min":1.492},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1479999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.2969999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.236},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.163,"15min":1.224},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.3799999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.93600000000000005}]},{"osd":7,"last update":"Sat Sep 30 00:37:09 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.159,"15min":1.23},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.323,"5min":2.851,"15min":2.851},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.24,"15min":1.3049999999999999},"min":{"1min":0.96699999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5029999999999999,"5min":1.887,"15min":1.887},"last":1.345}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796292,"kb_used_data":92903096,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903088,"statfs":{"total":95948165120,"available":95132762112,"internally_reserved":8192,"allocated":95132770304,"data_stored":95132770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48990,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66900000000000004,"15min":0.66100000000000003},"min":{"1min":0.49299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1890000000000001,"5min":1.345,"15min":1.345},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76900000000000002,"15min":0.72699999999999998},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0269999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.94499999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.54400000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.77700000000000002},"min":{"1min":0.52800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.016,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.1459999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.81200000000000006,"15min":0.80000000000000004},"min":{"1min":0.57699999999999996,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.099,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.80200000000000005,"15min":0.78200000000000003},"min":{"1min":0.55700000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70999999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.76200000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.373,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0109999999999999,"15min":0.97699999999999998},"min":{"1min":0.77400000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.343,"5min":1.466,"15min":1.466},"last":1.004}]},{"osd":5,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0489999999999999,"15min":1.0369999999999999},"min":{"1min":0.77100000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.538,"5min":2.169,"15min":2.169},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0840000000000001,"15min":1.0600000000000001},"min":{"1min":0.94999999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.375,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.109}]},{"osd":6,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.099,"15min":1.095},"min":{"1min":0.83599999999999997,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.472,"5min":1.474,"15min":1.474},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.1459999999999999,"15min":1.1259999999999999},"min":{"1min":1.052,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.46,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.0369999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.171,"15min":1.153},"min":{"1min":0.92800000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.4910000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.306},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1699999999999999,"15min":1.1659999999999999},"min":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.4259999999999999,"5min":1.554,"15min":1.554},"last":1.1870000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796348,"kb_used_data":92903048,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903032,"statfs":{"total":95948165120,"available":95132704768,"internally_reserved":16384,"allocated":95132721152,"data_stored":95132721152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53397,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72699999999999998,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":4.5780000000000003,"15min":4.5780000000000003},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82299999999999995,"15min":0.91700000000000004},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0449999999999999,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.89800000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.77500000000000002,"15min":0.81799999999999995},"min":{"1min":0.51600000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.0369999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.495,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.071,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.62}]},{"osd":3,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.878,"15min":0.91200000000000003},"min":{"1min":0.64600000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.177,"5min":1.49,"15min":1.49},"last":0.70099999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.998,"15min":1.0089999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.216,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0089999999999999,"15min":1.014},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1839999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.153}]},{"osd":5,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.008,"15min":1.004},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3140000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.454},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3859999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.105}]},{"osd":6,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.085,"15min":1.1220000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5569999999999999,"5min":1.607,"15min":1.607},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.3680000000000001,"15min":1.1719999999999999},"min":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3440000000000001,"5min":29.962,"15min":29.962},"last":1.337}]},{"osd":7,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1419999999999999,"15min":1.1779999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.6160000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1679999999999999},"min":{"1min":0.88,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.2050000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796332,"kb_used_data":92903064,"kb_used_omap":74,"kb_used_meta":0,"kb_avail":92903048,"statfs":{"total":95948165120,"available":95132721152,"internally_reserved":16384,"allocated":95132737536,"data_stored":95132737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75797,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.67500000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.98699999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73999999999999999,"15min":0.78600000000000003},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.0629999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.0629999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70299999999999996,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.90700000000000003,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.879},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3839999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.83399999999999996}]},{"osd":3,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85099999999999998,"15min":0.88600000000000001},"min":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2,"5min":1.282,"15min":1.282},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84299999999999997,"15min":0.85999999999999999},"min":{"1min":0.65700000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1120000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0049999999999999,"15min":1.0389999999999999},"min":{"1min":0.81699999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3100000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.018,"5min":0.999,"15min":1.006},"min":{"1min":0.73799999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3129999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.02}]},{"osd":5,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.014,"15min":1.0169999999999999},"min":{"1min":0.72999999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.1919999999999999,"5min":1.593,"15min":1.593},"last":1.157},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.88300000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.581,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.98499999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1279999999999999,"15min":1.1619999999999999},"min":{"1min":0.92800000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.4870000000000001,"5min":1.962,"15min":1.962},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.115,"15min":1.115},"min":{"1min":0.83599999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.3380000000000001,"5min":1.665,"15min":1.665},"last":1.224}]},{"osd":7,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1639999999999999,"15min":1.218},"min":{"1min":0.98199999999999998,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":1.111},{"interface":"front","average":{"1min":1.169,"5min":2.173,"15min":4.8449999999999998},"min":{"1min":0.88400000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.542,"5min":129.49100000000001,"15min":129.49100000000001},"last":1.3859999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796948,"kb_used_data":92902448,"kb_used_omap":129,"kb_used_meta":0,"kb_avail":92902432,"statfs":{"total":95948165120,"available":95132090368,"internally_reserved":16384,"allocated":95132106752,"data_stored":95132106752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":132236,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69399999999999995,"15min":0.72399999999999998},"min":{"1min":0.51200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.96499999999999997,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.745,"15min":0.70999999999999996},"min":{"1min":0.61499999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.82599999999999996}]},{"osd":2,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.746,"15min":0.752},"min":{"1min":0.59199999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.028,"5min":1.119,"15min":1.119},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.61699999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.75800000000000001,"15min":0.73499999999999999},"min":{"1min":0.56000000000000005,"5min":0.373,"15min":0.373},"max":{"1min":1.085,"5min":1.292,"15min":1.292},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88600000000000001,"15min":0.86599999999999999},"min":{"1min":0.57999999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2669999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.98199999999999998,"15min":0.97599999999999998},"min":{"1min":0.81999999999999995,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.395,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.032,"15min":1.0980000000000001},"min":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.1299999999999999,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.996,"15min":1.002},"min":{"1min":0.76600000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.26,"5min":2.165,"15min":2.165},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.151,"5min":1.1419999999999999,"15min":1.1439999999999999},"min":{"1min":0.78000000000000003,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.498,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.218}]},{"osd":6,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1399999999999999,"15min":1.115},"min":{"1min":0.83299999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.601,"5min":1.615,"15min":1.615},"last":1.167},{"interface":"front","average":{"1min":1.149,"5min":1.119,"15min":1.095},"min":{"1min":0.92100000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4410000000000001,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.0389999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:37:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1399999999999999,"15min":1.1299999999999999},"min":{"1min":0.80600000000000005,"5min":0.755,"15min":0.755},"max":{"1min":1.5409999999999999,"5min":1.71,"15min":1.71},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.129,"5min":2.7389999999999999,"15min":4.3449999999999998},"min":{"1min":0.94499999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.389,"5min":80.903000000000006,"15min":80.903000000000006},"last":1.0669999999999999}]}]}],"pool_statfs":[]}} 2023-09-30T00:38:11.289 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:38:11.574 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:11.574 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":50,"fsid":"df8ee23b-92e0-405f-810c-7a3d9ae641bd","created":"2023-09-30T00:32:52.458484+0000","modified":"2023-09-30T00:38:11.429677+0000","last_up_change":"2023-09-30T00:32:58.220653+0000","last_in_change":"2023-09-30T00:32:55.204071+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:32:58.672551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:33:03.847465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2a37e1e5-0043-46ce-9833-37acd6fd9caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":26915},{"type":"v1","addr":"172.21.15.110:6809","nonce":26915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":26915},{"type":"v1","addr":"172.21.15.110:6811","nonce":26915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":26915},{"type":"v1","addr":"172.21.15.110:6815","nonce":26915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":26915},{"type":"v1","addr":"172.21.15.110:6813","nonce":26915}]},"public_addr":"172.21.15.110:6809/26915","cluster_addr":"172.21.15.110:6811/26915","heartbeat_back_addr":"172.21.15.110:6815/26915","heartbeat_front_addr":"172.21.15.110:6813/26915","state":["exists","up"]},{"osd":1,"uuid":"71133118-837b-477d-8c8a-f0b6703508e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":26913},{"type":"v1","addr":"172.21.15.110:6801","nonce":26913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":26913},{"type":"v1","addr":"172.21.15.110:6803","nonce":26913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":26913},{"type":"v1","addr":"172.21.15.110:6807","nonce":26913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":26913},{"type":"v1","addr":"172.21.15.110:6805","nonce":26913}]},"public_addr":"172.21.15.110:6801/26913","cluster_addr":"172.21.15.110:6803/26913","heartbeat_back_addr":"172.21.15.110:6807/26913","heartbeat_front_addr":"172.21.15.110:6805/26913","state":["exists","up"]},{"osd":2,"uuid":"518bc2ab-0374-44c0-95e2-da0383144b93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":26926},{"type":"v1","addr":"172.21.15.110:6825","nonce":26926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":26926},{"type":"v1","addr":"172.21.15.110:6827","nonce":26926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":26926},{"type":"v1","addr":"172.21.15.110:6831","nonce":26926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":26926},{"type":"v1","addr":"172.21.15.110:6829","nonce":26926}]},"public_addr":"172.21.15.110:6825/26926","cluster_addr":"172.21.15.110:6827/26926","heartbeat_back_addr":"172.21.15.110:6831/26926","heartbeat_front_addr":"172.21.15.110:6829/26926","state":["exists","up"]},{"osd":3,"uuid":"e22cf827-1bef-4184-8575-54d91d8435c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":26914},{"type":"v1","addr":"172.21.15.110:6817","nonce":26914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":26914},{"type":"v1","addr":"172.21.15.110:6819","nonce":26914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":26914},{"type":"v1","addr":"172.21.15.110:6823","nonce":26914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":26914},{"type":"v1","addr":"172.21.15.110:6821","nonce":26914}]},"public_addr":"172.21.15.110:6817/26914","cluster_addr":"172.21.15.110:6819/26914","heartbeat_back_addr":"172.21.15.110:6823/26914","heartbeat_front_addr":"172.21.15.110:6821/26914","state":["exists","up"]},{"osd":4,"uuid":"0c705242-5495-4641-9a0c-011a1766dc49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":27048},{"type":"v1","addr":"172.21.15.157:6801","nonce":27048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":27048},{"type":"v1","addr":"172.21.15.157:6803","nonce":27048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":27048},{"type":"v1","addr":"172.21.15.157:6807","nonce":27048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":27048},{"type":"v1","addr":"172.21.15.157:6805","nonce":27048}]},"public_addr":"172.21.15.157:6801/27048","cluster_addr":"172.21.15.157:6803/27048","heartbeat_back_addr":"172.21.15.157:6807/27048","heartbeat_front_addr":"172.21.15.157:6805/27048","state":["exists","up"]},{"osd":5,"uuid":"9f973d43-09c3-4fed-ad02-8f208c64d9f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6823","nonce":27059},{"type":"v1","addr":"172.21.15.157:6825","nonce":27059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":27059},{"type":"v1","addr":"172.21.15.157:6827","nonce":27059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":27059},{"type":"v1","addr":"172.21.15.157:6831","nonce":27059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":27059},{"type":"v1","addr":"172.21.15.157:6829","nonce":27059}]},"public_addr":"172.21.15.157:6825/27059","cluster_addr":"172.21.15.157:6827/27059","heartbeat_back_addr":"172.21.15.157:6831/27059","heartbeat_front_addr":"172.21.15.157:6829/27059","state":["exists","up"]},{"osd":6,"uuid":"b9296c08-c456-43b4-882a-28b4979ea6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":27049},{"type":"v1","addr":"172.21.15.157:6817","nonce":27049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":27049},{"type":"v1","addr":"172.21.15.157:6819","nonce":27049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":27049},{"type":"v1","addr":"172.21.15.157:6824","nonce":27049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":27049},{"type":"v1","addr":"172.21.15.157:6821","nonce":27049}]},"public_addr":"172.21.15.157:6817/27049","cluster_addr":"172.21.15.157:6819/27049","heartbeat_back_addr":"172.21.15.157:6824/27049","heartbeat_front_addr":"172.21.15.157:6821/27049","state":["exists","up"]},{"osd":7,"uuid":"4e612da4-22aa-46e1-8324-d9982302791b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":27047},{"type":"v1","addr":"172.21.15.157:6809","nonce":27047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":27047},{"type":"v1","addr":"172.21.15.157:6811","nonce":27047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":27047},{"type":"v1","addr":"172.21.15.157:6815","nonce":27047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":27047},{"type":"v1","addr":"172.21.15.157:6813","nonce":27047}]},"public_addr":"172.21.15.157:6809/27047","cluster_addr":"172.21.15.157:6811/27047","heartbeat_back_addr":"172.21.15.157:6815/27047","heartbeat_front_addr":"172.21.15.157:6813/27047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:55.154980+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:01.206226+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:11.360802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:11.396380+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:35.216655+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:37:50.598288+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:06.008222+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:38:07.137400+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:38:12.589 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T00:38:12.589 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:12.751 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:12.752 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:12.752 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:12.765 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T00:38:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T00:38:13.069 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T00:38:13.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:13.214 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:13.215 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:13.215 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:13.226 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T00:38:13.520 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T00:38:13.534 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T00:38:13.534 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:13.675 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:13.675 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:13.675 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:13.687 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T00:38:13.994 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T00:38:14.009 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T00:38:14.009 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:14.150 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:14.150 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:14.150 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:14.162 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T00:38:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T00:38:14.471 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T00:38:14.471 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:14.610 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:14.611 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:14.611 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:14.623 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T00:38:14.914 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T00:38:14.924 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T00:38:14.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:15.069 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:15.070 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:15.070 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:15.082 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T00:38:15.366 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T00:38:15.376 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T00:38:15.376 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:15.510 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:15.511 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:15.511 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:15.523 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T00:38:15.811 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T00:38:15.821 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T00:38:15.821 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:38:15.959 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-09-30T00:38:15.960 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:38:15.960 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-09-30T00:38:15.971 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T00:38:16.267 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T00:38:16.277 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:38:16.562 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:16.564 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:38:16.576 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2023-09-30T00:38:14.701105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6371592,"kb_used_data":743223528,"kb_used_omap":596,"kb_used_meta":0,"kb_avail":743223448,"statfs":{"total":767585320960,"available":761060810752,"internally_reserved":81920,"allocated":761060892672,"data_stored":761060892672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":610491,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002240"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":70,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-09-30T00:38:08.806931+0000","last_change":"2023-09-30T00:37:00.203398+0000","last_active":"2023-09-30T00:38:08.806931+0000","last_peered":"2023-09-30T00:38:08.806931+0000","last_clean":"2023-09-30T00:38:08.806931+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:38:08.806931+0000","last_undegraded":"2023-09-30T00:38:08.806931+0000","last_fullsized":"2023-09-30T00:38:08.806931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:37:00.203204+0000","last_clean_scrub_stamp":"2023-09-30T00:37:00.203204+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:21.467829+0000","scrub_duration":0.20409011799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:11.640944+0000","last_change":"2023-09-30T00:37:15.055387+0000","last_active":"2023-09-30T00:38:11.640944+0000","last_peered":"2023-09-30T00:38:11.640944+0000","last_clean":"2023-09-30T00:38:11.640944+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:38:11.640944+0000","last_undegraded":"2023-09-30T00:38:11.640944+0000","last_fullsized":"2023-09-30T00:38:11.640944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:15.055188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:15.055188+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.104113765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":71,"reported_epoch":50,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-30T00:38:12.112288+0000","last_change":"2023-09-30T00:38:13.856066+0000","last_active":"2023-09-30T00:38:12.112288+0000","last_peered":"2023-09-30T00:38:12.112288+0000","last_clean":"2023-09-30T00:38:12.112288+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:38:12.112288+0000","last_undegraded":"2023-09-30T00:38:12.112288+0000","last_fullsized":"2023-09-30T00:38:12.112288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:36:59.844873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:49.776230+0000","last_clean_scrub_stamp":"2023-09-30T00:36:59.844873+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10427676600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":71,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:11.540867+0000","last_change":"2023-09-30T00:37:07.344322+0000","last_active":"2023-09-30T00:38:11.540867+0000","last_peered":"2023-09-30T00:38:11.540867+0000","last_clean":"2023-09-30T00:38:11.540867+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:38:11.540867+0000","last_undegraded":"2023-09-30T00:38:11.540867+0000","last_fullsized":"2023-09-30T00:38:11.540867+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:07.344143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:07.344143+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.40480254500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":69,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-09-30T00:38:07.425252+0000","last_change":"2023-09-30T00:37:11.469764+0000","last_active":"2023-09-30T00:38:07.425252+0000","last_peered":"2023-09-30T00:38:07.425252+0000","last_clean":"2023-09-30T00:38:07.425252+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:38:07.425252+0000","last_undegraded":"2023-09-30T00:38:07.425252+0000","last_fullsized":"2023-09-30T00:38:07.425252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:11.469587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:33:04.279938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:11.469587+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:20.615198+0000","scrub_duration":0.204916511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-09-30T00:38:08.726940+0000","last_change":"2023-09-30T00:37:12.329649+0000","last_active":"2023-09-30T00:38:08.726940+0000","last_peered":"2023-09-30T00:38:08.726940+0000","last_clean":"2023-09-30T00:38:08.726940+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:38:08.726940+0000","last_undegraded":"2023-09-30T00:38:08.726940+0000","last_fullsized":"2023-09-30T00:38:08.726940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:37:12.329539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:34:38.283938+0000","last_clean_scrub_stamp":"2023-09-30T00:37:12.329539+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:38:35.247630+0000","scrub_duration":0.002197844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-09-30T00:38:09.925381+0000","last_change":"2023-09-30T00:38:09.925292+0000","last_active":"2023-09-30T00:38:09.925381+0000","last_peered":"2023-09-30T00:38:09.925381+0000","last_clean":"2023-09-30T00:38:09.925381+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:38:09.925381+0000","last_undegraded":"2023-09-30T00:38:09.925381+0000","last_fullsized":"2023-09-30T00:38:09.925381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:09.925147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:35:38.924322+0000","last_clean_scrub_stamp":"2023-09-30T00:38:09.925147+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:29.168244+0000","scrub_duration":0.0035836010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":80,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-09-30T00:38:12.485235+0000","last_change":"2023-09-30T00:38:12.485133+0000","last_active":"2023-09-30T00:38:12.485235+0000","last_peered":"2023-09-30T00:38:12.485235+0000","last_clean":"2023-09-30T00:38:12.485235+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:38:12.485235+0000","last_undegraded":"2023-09-30T00:38:12.485235+0000","last_fullsized":"2023-09-30T00:38:12.485235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:12.484964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:12.484964+0000","last_clean_scrub_stamp":"2023-09-30T00:38:12.484964+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:25.373198+0000","scrub_duration":0.20483620899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":233,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-09-30T00:38:08.907387+0000","last_change":"2023-09-30T00:38:02.085975+0000","last_active":"2023-09-30T00:38:08.907387+0000","last_peered":"2023-09-30T00:38:08.907387+0000","last_clean":"2023-09-30T00:38:08.907387+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:38:08.907387+0000","last_undegraded":"2023-09-30T00:38:08.907387+0000","last_fullsized":"2023-09-30T00:38:08.907387+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-09-30T00:38:02.085822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:32:59.239112+0000","last_clean_scrub_stamp":"2023-09-30T00:38:02.085822+0000","objects_scrubbed":2,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:17.769551+0000","scrub_duration":0.20558635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796888,"kb_used_data":92902500,"kb_used_omap":141,"kb_used_meta":0,"kb_avail":92902492,"statfs":{"total":95948165120,"available":95132151808,"internally_reserved":8192,"allocated":95132160000,"data_stored":95132160000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144641,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.91700000000000004,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.069},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.0589999999999999,"15min":1.5129999999999999},"min":{"1min":0.57899999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":18.120000000000001,"15min":18.120000000000001},"last":0.97499999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90200000000000002,"15min":0.92600000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.93600000000000005,"15min":0.95699999999999996},"min":{"1min":0.71999999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.3999999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88100000000000001,"15min":0.85499999999999998},"min":{"1min":0.58399999999999996,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":1.355},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.97199999999999998,"15min":0.98299999999999998},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":0.90000000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95199999999999996,"15min":0.97599999999999998},"min":{"1min":0.73799999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.02,"15min":1.044},"min":{"1min":0.73299999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.27}]},{"osd":4,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.64800000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.028,"15min":1.0449999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.45}]},{"osd":5,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.997,"15min":1.022},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.92200000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":1.2050000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.075,"15min":1.115},"min":{"1min":0.85499999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.391,"5min":1.45,"15min":1.45},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.139,"15min":1.1679999999999999},"min":{"1min":0.77600000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.498}]}]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796312,"kb_used_data":92903076,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92903068,"statfs":{"total":95948165120,"available":95132741632,"internally_reserved":8192,"allocated":95132749824,"data_stored":95132749824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57980,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.90800000000000003,"15min":0.89600000000000002},"min":{"1min":0.65800000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2709999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91200000000000003,"15min":0.98399999999999999},"min":{"1min":0.71699999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":1.151}]},{"osd":1,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.64500000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.23,"5min":1.351,"15min":1.351},"last":1.234},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.95599999999999996,"15min":0.96799999999999997},"min":{"1min":0.626,"5min":0.621,"15min":0.621},"max":{"1min":1.2490000000000001,"5min":1.341,"15min":1.341},"last":0.752}]},{"osd":2,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.96599999999999997},"min":{"1min":0.69199999999999995,"5min":0.621,"15min":0.621},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.403},{"interface":"front","average":{"1min":0.995,"5min":1.002,"15min":1.0580000000000001},"min":{"1min":0.68999999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.304,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.745,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.012,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0580000000000001,"15min":1.1259999999999999},"min":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3180000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.101}]},{"osd":4,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.97299999999999998,"15min":1.081},"min":{"1min":0.68500000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.248,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0840000000000001,"15min":1.1639999999999999},"min":{"1min":0.92200000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.325,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.4450000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0509999999999999,"15min":1.137},"min":{"1min":0.68300000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1950000000000001,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.169,"15min":1.2410000000000001},"min":{"1min":0.753,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3600000000000001,"5min":1.627,"15min":1.627},"last":1.3280000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.173,"15min":1.2430000000000001},"min":{"1min":0.90500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.3540000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":1.272},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.333,"15min":1.855},"min":{"1min":0.89600000000000002,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.3620000000000001,"5min":16.701000000000001,"15min":16.701000000000001},"last":1.02}]}]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796328,"kb_used_data":92903068,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903052,"statfs":{"total":95948165120,"available":95132725248,"internally_reserved":16384,"allocated":95132741632,"data_stored":95132741632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54400,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91900000000000004,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2390000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.89700000000000002,"15min":0.878},"min":{"1min":0.55200000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.75600000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":0.871},"min":{"1min":0.63900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.405,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93600000000000005,"15min":0.91000000000000003},"min":{"1min":0.72999999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.79500000000000004}]},{"osd":2,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.89600000000000002},"min":{"1min":0.59399999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98999999999999999,"15min":0.96699999999999997},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.351,"5min":1.546,"15min":1.546},"last":0.84099999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.69699999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":1.091},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":1.0249999999999999},"min":{"1min":0.76600000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3799999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.56999999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93700000000000006,"15min":0.96299999999999997},"min":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.24,"5min":1.425,"15min":1.425},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.91300000000000003}]},{"osd":6,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0589999999999999,"15min":1.0840000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.90200000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4750000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.1659999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4730000000000001,"5min":1.514,"15min":1.514},"last":1.014},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3759999999999999},"min":{"1min":0.93300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6519999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":1.1259999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796200,"kb_used_data":92903188,"kb_used_omap":39,"kb_used_meta":0,"kb_avail":92903180,"statfs":{"total":95948165120,"available":95132856320,"internally_reserved":8192,"allocated":95132864512,"data_stored":95132864512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40196,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92600000000000005,"15min":0.97899999999999998},"min":{"1min":0.64400000000000002,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3120000000000001,"5min":1.446,"15min":1.446},"last":1.3120000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.874,"15min":0.92100000000000004},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.399,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.67000000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.58999999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.407,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.147,"5min":1.0249999999999999,"15min":1.139},"min":{"1min":0.79500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.1190000000000002,"5min":5.1319999999999997,"15min":5.1319999999999997},"last":1.2270000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.92900000000000005,"15min":0.94499999999999995},"min":{"1min":0.752,"5min":0.622,"15min":0.622},"max":{"1min":1.167,"5min":1.456,"15min":1.456},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.98099999999999998,"15min":1.0449999999999999},"min":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.72499999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.97599999999999998,"15min":1.046},"min":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.298,"5min":2.048,"15min":2.048},"last":1.034},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.032,"15min":1.0620000000000001},"min":{"1min":0.70599999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.4139999999999999,"5min":1.571,"15min":1.571},"last":0.97399999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":1.0049999999999999},"min":{"1min":0.61899999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.518,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1020000000000001,"15min":1.1000000000000001},"min":{"1min":0.91800000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":1.198}]},{"osd":6,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0740000000000001,"15min":1.143},"min":{"1min":0.80200000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.306,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.157},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.159,"15min":1.2230000000000001},"min":{"1min":0.89800000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.651,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.3799999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.131,"15min":1.2210000000000001},"min":{"1min":0.76800000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.635,"5min":2.851,"15min":2.851},"last":1.266},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.236,"15min":1.304},"min":{"1min":1.133,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.571,"5min":1.887,"15min":1.887},"last":1.2849999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796312,"kb_used_data":92903076,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92903068,"statfs":{"total":95948165120,"available":95132741632,"internally_reserved":8192,"allocated":95132749824,"data_stored":95132749824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49116,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67800000000000005,"15min":0.66400000000000003},"min":{"1min":0.52200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2929999999999999,"5min":1.345,"15min":1.345},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78600000000000003,"15min":0.73199999999999998},"min":{"1min":0.56000000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.004,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0049999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.81899999999999995,"15min":0.78500000000000003},"min":{"1min":0.623,"5min":0.372,"15min":0.372},"max":{"1min":1.3859999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82299999999999995,"15min":0.80300000000000005},"min":{"1min":0.61799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2090000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81899999999999995,"15min":0.78700000000000003},"min":{"1min":0.56999999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.70999999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0109999999999999,"15min":1},"min":{"1min":0.76800000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.351,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.073},{"interface":"front","average":{"1min":1.05,"5min":1.0289999999999999,"15min":0.98299999999999998},"min":{"1min":0.82099999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.411,"5min":1.466,"15min":1.466},"last":1.2190000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.056,"15min":1.0389999999999999},"min":{"1min":0.877,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.5369999999999999,"5min":2.169,"15min":2.169},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.081,"15min":1.0589999999999999},"min":{"1min":0.78800000000000003,"5min":0.752,"15min":0.752},"max":{"1min":1.2909999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.0469999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1100000000000001,"15min":1.0980000000000001},"min":{"1min":0.91300000000000003,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.143},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.155,"15min":1.129},"min":{"1min":0.83399999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.4650000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.121}]},{"osd":7,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.179,"15min":1.155},"min":{"1min":0.88,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.5640000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.175,"15min":1.167},"min":{"1min":0.94699999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.464,"5min":1.554,"15min":1.554},"last":1.2789999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796320,"kb_used_data":92903076,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903060,"statfs":{"total":95948165120,"available":95132733440,"internally_reserved":16384,"allocated":95132749824,"data_stored":95132749824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54717,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72699999999999998,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":4.5780000000000003,"15min":4.5780000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82299999999999995,"15min":0.91700000000000004},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0449999999999999,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.77500000000000002,"15min":0.81799999999999995},"min":{"1min":0.51600000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.0369999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.495,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.071,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.875,"5min":0.878,"15min":0.91200000000000003},"min":{"1min":0.64600000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.177,"5min":1.49,"15min":1.49},"last":0.996}]},{"osd":4,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.998,"15min":1.0089999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.216,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0089999999999999,"15min":1.014},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1839999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.97799999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.008,"15min":1.004},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3140000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3859999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.0760000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.085,"15min":1.1220000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5569999999999999,"5min":1.607,"15min":1.607},"last":1.198},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.3680000000000001,"15min":1.1719999999999999},"min":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3440000000000001,"5min":29.962,"15min":29.962},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1419999999999999,"15min":1.1779999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.6160000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1679999999999999},"min":{"1min":0.88,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.1599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796320,"kb_used_data":92903068,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903060,"statfs":{"total":95948165120,"available":95132733440,"internally_reserved":8192,"allocated":95132741632,"data_stored":95132741632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77117,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.67500000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.98699999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73999999999999999,"15min":0.78600000000000003},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.0629999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70299999999999996,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.90700000000000003,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3839999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85099999999999998,"15min":0.88600000000000001},"min":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2,"5min":1.282,"15min":1.282},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84299999999999997,"15min":0.85999999999999999},"min":{"1min":0.65700000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1120000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.91500000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0049999999999999,"15min":1.0389999999999999},"min":{"1min":0.81699999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3100000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.018,"5min":0.999,"15min":1.006},"min":{"1min":0.73799999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3129999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.411}]},{"osd":5,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.014,"15min":1.0169999999999999},"min":{"1min":0.72999999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.1919999999999999,"5min":1.593,"15min":1.593},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.88300000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.581,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.5309999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1279999999999999,"15min":1.1619999999999999},"min":{"1min":0.92800000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.4870000000000001,"5min":1.962,"15min":1.962},"last":1.141},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.115,"15min":1.115},"min":{"1min":0.83599999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.3380000000000001,"5min":1.665,"15min":1.665},"last":1.504}]},{"osd":7,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1639999999999999,"15min":1.218},"min":{"1min":0.98199999999999998,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":1.379},{"interface":"front","average":{"1min":1.169,"5min":2.173,"15min":4.8449999999999998},"min":{"1min":0.88400000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.542,"5min":129.49100000000001,"15min":129.49100000000001},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796912,"kb_used_data":92902476,"kb_used_omap":129,"kb_used_meta":0,"kb_avail":92902468,"statfs":{"total":95948165120,"available":95132127232,"internally_reserved":8192,"allocated":95132135424,"data_stored":95132135424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":132324,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68400000000000005,"15min":0.71999999999999997},"min":{"1min":0.51200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.99099999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.754,"15min":0.71299999999999997},"min":{"1min":0.442,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1100000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.73499999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.73499999999999999,"15min":0.749},"min":{"1min":0.50800000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":1.119,"15min":1.119},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.006,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.95899999999999996}]},{"osd":3,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.77900000000000003,"15min":0.74199999999999999},"min":{"1min":0.56699999999999995,"5min":0.373,"15min":0.373},"max":{"1min":1.1870000000000001,"5min":1.292,"15min":1.292},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.88600000000000001,"15min":0.86599999999999999},"min":{"1min":0.64000000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.038,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":1.022}]},{"osd":4,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.97999999999999998,"15min":0.97599999999999998},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.1599999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.079},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.997,"15min":1.0860000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.1970000000000001,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":0.98999999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.79700000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.411,"5min":2.165,"15min":2.165},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1259999999999999,"15min":1.139},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.4079999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.1240000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.137,"15min":1.1140000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.347,"5min":1.615,"15min":1.615},"last":1.266},{"interface":"front","average":{"1min":1.083,"5min":1.119,"15min":1.095},"min":{"1min":0.81799999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.327,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.157}]},{"osd":7,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.91400000000000003,"5min":0.755,"15min":0.755},"max":{"1min":1.3149999999999999,"5min":1.71,"15min":1.71},"last":1.228},{"interface":"front","average":{"1min":1.157,"5min":1.9399999999999999,"15min":4.0789999999999997},"min":{"1min":0.94499999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.486,"5min":80.903000000000006,"15min":80.903000000000006},"last":1.1879999999999999}]}]}],"pool_statfs":[]}} 2023-09-30T00:38:16.581 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T00:37:00.203204+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=37, tm_sec=0, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.581 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T00:37:15.055188+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=37, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.581 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T00:36:59.844873+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=36, tm_sec=59, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.582 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T00:37:07.344143+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=37, tm_sec=7, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.582 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T00:37:11.469587+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=37, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.582 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T00:37:12.329539+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=37, tm_sec=12, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.582 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T00:38:09.925147+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=9, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.582 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T00:38:02.085822+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=38, tm_sec=11, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:38:16.582 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T00:38:34.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-09-30T00:38:34.630+0000 7f5b239fc700 -1 osd.3 50 handle_osd_map injecting map crc failure 2023-09-30T00:38:36.583 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:38:36.878 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T00:38:36.880 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-09-30T00:38:36.892 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2023-09-30T00:38:36.707541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6371192,"kb_used_data":743223848,"kb_used_omap":624,"kb_used_meta":0,"kb_avail":743223848,"statfs":{"total":767585320960,"available":761061220352,"internally_reserved":0,"allocated":761061220352,"data_stored":761061220352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":639021,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002922"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":81,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:17.613629+0000","last_change":"2023-09-30T00:38:17.613510+0000","last_active":"2023-09-30T00:38:17.613629+0000","last_peered":"2023-09-30T00:38:17.613629+0000","last_clean":"2023-09-30T00:38:17.613629+0000","last_became_active":"2023-09-30T00:33:05.406510+0000","last_became_peered":"2023-09-30T00:33:05.406510+0000","last_unstale":"2023-09-30T00:38:17.613629+0000","last_undegraded":"2023-09-30T00:38:17.613629+0000","last_fullsized":"2023-09-30T00:38:17.613629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:17.613349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:17.613349+0000","last_clean_scrub_stamp":"2023-09-30T00:38:17.613349+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:42.911300+0000","scrub_duration":0.30420523199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:17.231572+0000","last_change":"2023-09-30T00:38:17.231504+0000","last_active":"2023-09-30T00:38:17.231572+0000","last_peered":"2023-09-30T00:38:17.231572+0000","last_clean":"2023-09-30T00:38:17.231572+0000","last_became_active":"2023-09-30T00:33:05.303639+0000","last_became_peered":"2023-09-30T00:33:05.303639+0000","last_unstale":"2023-09-30T00:38:17.231572+0000","last_undegraded":"2023-09-30T00:38:17.231572+0000","last_fullsized":"2023-09-30T00:38:17.231572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:17.231347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:17.231347+0000","last_clean_scrub_stamp":"2023-09-30T00:38:17.231347+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:30.620163+0000","scrub_duration":0.30431612400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:30.061620+0000","last_change":"2023-09-30T00:38:30.061550+0000","last_active":"2023-09-30T00:38:30.061620+0000","last_peered":"2023-09-30T00:38:30.061620+0000","last_clean":"2023-09-30T00:38:30.061620+0000","last_became_active":"2023-09-30T00:33:05.306205+0000","last_became_peered":"2023-09-30T00:33:05.306205+0000","last_unstale":"2023-09-30T00:38:30.061620+0000","last_undegraded":"2023-09-30T00:38:30.061620+0000","last_fullsized":"2023-09-30T00:38:30.061620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:30.061407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:30.061407+0000","last_clean_scrub_stamp":"2023-09-30T00:38:30.061407+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:34.401567+0000","scrub_duration":0.204277078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:16.198691+0000","last_change":"2023-09-30T00:38:16.198623+0000","last_active":"2023-09-30T00:38:16.198691+0000","last_peered":"2023-09-30T00:38:16.198691+0000","last_clean":"2023-09-30T00:38:16.198691+0000","last_became_active":"2023-09-30T00:33:05.306020+0000","last_became_peered":"2023-09-30T00:33:05.306020+0000","last_unstale":"2023-09-30T00:38:16.198691+0000","last_undegraded":"2023-09-30T00:38:16.198691+0000","last_fullsized":"2023-09-30T00:38:16.198691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:16.198480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:16.198480+0000","last_clean_scrub_stamp":"2023-09-30T00:38:16.198480+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:29.087040+0000","scrub_duration":0.30357001300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:21.553217+0000","last_change":"2023-09-30T00:38:21.553078+0000","last_active":"2023-09-30T00:38:21.553217+0000","last_peered":"2023-09-30T00:38:21.553217+0000","last_clean":"2023-09-30T00:38:21.553217+0000","last_became_active":"2023-09-30T00:33:05.304727+0000","last_became_peered":"2023-09-30T00:33:05.304727+0000","last_unstale":"2023-09-30T00:38:21.553217+0000","last_undegraded":"2023-09-30T00:38:21.553217+0000","last_fullsized":"2023-09-30T00:38:21.553217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:21.552870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:21.552870+0000","last_clean_scrub_stamp":"2023-09-30T00:38:21.552870+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:29.976090+0000","scrub_duration":0.30418483200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:17.958459+0000","last_change":"2023-09-30T00:38:17.958386+0000","last_active":"2023-09-30T00:38:17.958459+0000","last_peered":"2023-09-30T00:38:17.958459+0000","last_clean":"2023-09-30T00:38:17.958459+0000","last_became_active":"2023-09-30T00:33:05.430636+0000","last_became_peered":"2023-09-30T00:33:05.430636+0000","last_unstale":"2023-09-30T00:38:17.958459+0000","last_undegraded":"2023-09-30T00:38:17.958459+0000","last_fullsized":"2023-09-30T00:38:17.958459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:17.958236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:17.958236+0000","last_clean_scrub_stamp":"2023-09-30T00:38:17.958236+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:20.563586+0000","scrub_duration":0.304144362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:23.010721+0000","last_change":"2023-09-30T00:38:23.010663+0000","last_active":"2023-09-30T00:38:23.010721+0000","last_peered":"2023-09-30T00:38:23.010721+0000","last_clean":"2023-09-30T00:38:23.010721+0000","last_became_active":"2023-09-30T00:33:05.700820+0000","last_became_peered":"2023-09-30T00:33:05.700820+0000","last_unstale":"2023-09-30T00:38:23.010721+0000","last_undegraded":"2023-09-30T00:38:23.010721+0000","last_fullsized":"2023-09-30T00:38:23.010721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:23.010505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:23.010505+0000","last_clean_scrub_stamp":"2023-09-30T00:38:23.010505+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:43.085452+0000","scrub_duration":0.10416486699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:16.307985+0000","last_change":"2023-09-30T00:38:16.307848+0000","last_active":"2023-09-30T00:38:16.307985+0000","last_peered":"2023-09-30T00:38:16.307985+0000","last_clean":"2023-09-30T00:38:16.307985+0000","last_became_active":"2023-09-30T00:33:05.303556+0000","last_became_peered":"2023-09-30T00:33:05.303556+0000","last_unstale":"2023-09-30T00:38:16.307985+0000","last_undegraded":"2023-09-30T00:38:16.307985+0000","last_fullsized":"2023-09-30T00:38:16.307985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:38:16.307685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:38:16.307685+0000","last_clean_scrub_stamp":"2023-09-30T00:38:16.307685+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:36.792633+0000","scrub_duration":0.103851158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"50'119","reported_seq":246,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-09-30T00:38:18.480155+0000","last_change":"2023-09-30T00:38:18.480082+0000","last_active":"2023-09-30T00:38:18.480155+0000","last_peered":"2023-09-30T00:38:18.480155+0000","last_clean":"2023-09-30T00:38:18.480155+0000","last_became_active":"2023-09-30T00:33:00.377326+0000","last_became_peered":"2023-09-30T00:33:00.377326+0000","last_unstale":"2023-09-30T00:38:18.480155+0000","last_undegraded":"2023-09-30T00:38:18.480155+0000","last_fullsized":"2023-09-30T00:38:18.480155+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'119","last_scrub_stamp":"2023-09-30T00:38:18.479920+0000","last_deep_scrub":"50'119","last_deep_scrub_stamp":"2023-09-30T00:38:18.479920+0000","last_clean_scrub_stamp":"2023-09-30T00:38:18.479920+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T00:39:24.228364+0000","scrub_duration":0.50996556000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796840,"kb_used_data":92902540,"kb_used_omap":148,"kb_used_meta":0,"kb_avail":92902540,"statfs":{"total":95948165120,"available":95132200960,"internally_reserved":0,"allocated":95132200960,"data_stored":95132200960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":152540,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.91700000000000004,"15min":0.873},"min":{"1min":0.64200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.0589999999999999,"15min":1.5129999999999999},"min":{"1min":0.57899999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":18.120000000000001,"15min":18.120000000000001},"last":0.68899999999999995}]},{"osd":1,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90200000000000002,"15min":0.92600000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.93600000000000005,"15min":0.95699999999999996},"min":{"1min":0.71999999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.98099999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.88100000000000001,"15min":0.85499999999999998},"min":{"1min":0.58399999999999996,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.97199999999999998,"15min":0.98299999999999998},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":0.88}]},{"osd":3,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95199999999999996,"15min":0.97599999999999998},"min":{"1min":0.73799999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.02,"15min":1.044},"min":{"1min":0.73299999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.1080000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.64800000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.028,"15min":1.0449999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.90300000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.997,"15min":1.022},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.202},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.92200000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":1.1339999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.075,"15min":1.115},"min":{"1min":0.85499999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.391,"5min":1.45,"15min":1.45},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.139,"15min":1.1679999999999999},"min":{"1min":0.77600000000000002,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.2509999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796264,"kb_used_data":92903116,"kb_used_omap":59,"kb_used_meta":0,"kb_avail":92903116,"statfs":{"total":95948165120,"available":95132790784,"internally_reserved":0,"allocated":95132790784,"data_stored":95132790784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60471,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.90800000000000003,"15min":0.89600000000000002},"min":{"1min":0.65800000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2709999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91200000000000003,"15min":0.98399999999999999},"min":{"1min":0.71699999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.745}]},{"osd":1,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.64500000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.23,"5min":1.351,"15min":1.351},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.95599999999999996,"15min":0.96799999999999997},"min":{"1min":0.626,"5min":0.621,"15min":0.621},"max":{"1min":1.2490000000000001,"5min":1.341,"15min":1.341},"last":1.151}]},{"osd":2,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95699999999999996,"15min":0.96599999999999997},"min":{"1min":0.69199999999999995,"5min":0.621,"15min":0.621},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.995,"5min":1.002,"15min":1.0580000000000001},"min":{"1min":0.68999999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.304,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.90600000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.745,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.012,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0580000000000001,"15min":1.1259999999999999},"min":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3180000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.97899999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.97299999999999998,"15min":1.081},"min":{"1min":0.68500000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.248,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0840000000000001,"15min":1.1639999999999999},"min":{"1min":0.92200000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.325,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.2509999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0509999999999999,"15min":1.137},"min":{"1min":0.68300000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1950000000000001,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.169,"15min":1.2410000000000001},"min":{"1min":0.753,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3600000000000001,"5min":1.627,"15min":1.627},"last":1.1899999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:07 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.173,"15min":1.2430000000000001},"min":{"1min":0.90500000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.3540000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.333,"15min":1.855},"min":{"1min":0.89600000000000002,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.3620000000000001,"5min":16.701000000000001,"15min":16.701000000000001},"last":1.121}]}]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796272,"kb_used_data":92903108,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92903108,"statfs":{"total":95948165120,"available":95132782592,"internally_reserved":0,"allocated":95132782592,"data_stored":95132782592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56868,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91900000000000004,"15min":0.89500000000000002},"min":{"1min":0.624,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2390000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.89700000000000002,"15min":0.878},"min":{"1min":0.55200000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.198,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.872}]},{"osd":1,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":0.871},"min":{"1min":0.63900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.405,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93600000000000005,"15min":0.91000000000000003},"min":{"1min":0.72999999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.0569999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.93999999999999995,"15min":0.89600000000000002},"min":{"1min":0.59399999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98999999999999999,"15min":0.96699999999999997},"min":{"1min":0.71399999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.351,"5min":1.546,"15min":1.546},"last":0.99299999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.69699999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3320000000000001,"5min":1.421,"15min":1.421},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0449999999999999,"15min":1.0249999999999999},"min":{"1min":0.76600000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3799999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.095}]},{"osd":4,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.93700000000000006,"15min":0.96299999999999997},"min":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.24,"5min":1.425,"15min":1.425},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.3,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.173}]},{"osd":6,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0589999999999999,"15min":1.0840000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.026},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.90200000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4750000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.1279999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1120000000000001,"15min":1.1180000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4730000000000001,"5min":1.514,"15min":1.514},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3759999999999999},"min":{"1min":0.93300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6519999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":1.2130000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796152,"kb_used_data":92903228,"kb_used_omap":39,"kb_used_meta":0,"kb_avail":92903228,"statfs":{"total":95948165120,"available":95132905472,"internally_reserved":0,"allocated":95132905472,"data_stored":95132905472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40246,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92600000000000005,"15min":0.97899999999999998},"min":{"1min":0.64400000000000002,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3120000000000001,"5min":1.446,"15min":1.446},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.874,"15min":0.92100000000000004},"min":{"1min":0.626,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.399,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.71799999999999997}]},{"osd":1,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92500000000000004,"15min":0.95599999999999996},"min":{"1min":0.58999999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.407,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.0249999999999999,"15min":1.139},"min":{"1min":0.79500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.1190000000000002,"5min":5.1319999999999997,"15min":5.1319999999999997},"last":1.1519999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.92900000000000005,"15min":0.94499999999999995},"min":{"1min":0.752,"5min":0.622,"15min":0.622},"max":{"1min":1.167,"5min":1.456,"15min":1.456},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.98099999999999998,"15min":1.0449999999999999},"min":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":1.0369999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.97599999999999998,"15min":1.046},"min":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.298,"5min":2.048,"15min":2.048},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.032,"15min":1.0620000000000001},"min":{"1min":0.70599999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.4139999999999999,"5min":1.571,"15min":1.571},"last":1.0660000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":1.0049999999999999},"min":{"1min":0.61899999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.518,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1020000000000001,"15min":1.1000000000000001},"min":{"1min":0.91800000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.89600000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0740000000000001,"15min":1.143},"min":{"1min":0.80200000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.306,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.127},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.159,"15min":1.2230000000000001},"min":{"1min":0.89800000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.651,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.84199999999999997}]},{"osd":7,"last update":"Sat Sep 30 00:38:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.131,"15min":1.2210000000000001},"min":{"1min":0.76800000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.635,"5min":2.851,"15min":2.851},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.236,"15min":1.304},"min":{"1min":1.133,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.571,"5min":1.887,"15min":1.887},"last":0.98699999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796264,"kb_used_data":92903116,"kb_used_omap":49,"kb_used_meta":0,"kb_avail":92903116,"statfs":{"total":95948165120,"available":95132790784,"internally_reserved":0,"allocated":95132790784,"data_stored":95132790784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":50448,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67800000000000005,"15min":0.66400000000000003},"min":{"1min":0.52200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2929999999999999,"5min":1.345,"15min":1.345},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78600000000000003,"15min":0.73199999999999998},"min":{"1min":0.56000000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.004,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.752}]},{"osd":1,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72999999999999998},"min":{"1min":0.52500000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0049999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.81899999999999995,"15min":0.78500000000000003},"min":{"1min":0.623,"5min":0.372,"15min":0.372},"max":{"1min":1.3859999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.82599999999999996}]},{"osd":2,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82299999999999995,"15min":0.80300000000000005},"min":{"1min":0.61799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.2090000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.81899999999999995,"15min":0.78700000000000003},"min":{"1min":0.56999999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.92700000000000005}]},{"osd":4,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0109999999999999,"15min":1},"min":{"1min":0.76800000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.351,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.036},{"interface":"front","average":{"1min":1.05,"5min":1.0289999999999999,"15min":0.98299999999999998},"min":{"1min":0.82099999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.411,"5min":1.466,"15min":1.466},"last":1.0049999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.056,"15min":1.0389999999999999},"min":{"1min":0.877,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.5369999999999999,"5min":2.169,"15min":2.169},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.081,"15min":1.0589999999999999},"min":{"1min":0.78800000000000003,"5min":0.752,"15min":0.752},"max":{"1min":1.2909999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.147}]},{"osd":6,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1100000000000001,"15min":1.0980000000000001},"min":{"1min":0.91300000000000003,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.155,"15min":1.129},"min":{"1min":0.83399999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.4650000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.2969999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:12 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.179,"15min":1.155},"min":{"1min":0.88,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.5640000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.232},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.175,"15min":1.167},"min":{"1min":0.94699999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.464,"5min":1.554,"15min":1.554},"last":1.256}]}]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796264,"kb_used_data":92903116,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92903116,"statfs":{"total":95948165120,"available":95132790784,"internally_reserved":0,"allocated":95132790784,"data_stored":95132790784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57166,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.72699999999999998,"15min":0.81999999999999995},"min":{"1min":0.505,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":4.5780000000000003,"15min":4.5780000000000003},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.82299999999999995,"15min":0.91700000000000004},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0449999999999999,"5min":4.6130000000000004,"15min":4.6130000000000004},"last":0.873}]},{"osd":1,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.77500000000000002,"15min":0.81799999999999995},"min":{"1min":0.51600000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.0369999999999999,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.628},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.495,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.071,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.878,"15min":0.91200000000000003},"min":{"1min":0.64600000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.177,"5min":1.49,"15min":1.49},"last":0.94099999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.998,"15min":1.0089999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.216,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0089999999999999,"15min":1.014},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1839999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.155}]},{"osd":5,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.008,"15min":1.004},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3140000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.3859999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.1060000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.085,"15min":1.1220000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5569999999999999,"5min":1.607,"15min":1.607},"last":1.024},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.3680000000000001,"15min":1.1719999999999999},"min":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3440000000000001,"5min":29.962,"15min":29.962},"last":1.0629999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:05 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1419999999999999,"15min":1.1779999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.6160000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1679999999999999},"min":{"1min":0.88,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.129}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796272,"kb_used_data":92903108,"kb_used_omap":80,"kb_used_meta":0,"kb_avail":92903108,"statfs":{"total":95948165120,"available":95132782592,"internally_reserved":0,"allocated":95132782592,"data_stored":95132782592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":82049,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.67500000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.98699999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.73999999999999999,"15min":0.78600000000000003},"min":{"1min":0.51400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.0629999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.86899999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70299999999999996,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.90700000000000003,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.57499999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3839999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.76600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85099999999999998,"15min":0.88600000000000001},"min":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2,"5min":1.282,"15min":1.282},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84299999999999997,"15min":0.85999999999999999},"min":{"1min":0.65700000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1120000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0069999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0049999999999999,"15min":1.0389999999999999},"min":{"1min":0.81699999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3100000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.018,"5min":0.999,"15min":1.006},"min":{"1min":0.73799999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3129999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.0309999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.014,"15min":1.0169999999999999},"min":{"1min":0.72999999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.1919999999999999,"5min":1.593,"15min":1.593},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.88300000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.581,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.0900000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1279999999999999,"15min":1.1619999999999999},"min":{"1min":0.92800000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.4870000000000001,"5min":1.962,"15min":1.962},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.115,"15min":1.115},"min":{"1min":0.83599999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.3380000000000001,"5min":1.665,"15min":1.665},"last":1.1319999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:38:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1639999999999999,"15min":1.218},"min":{"1min":0.98199999999999998,"5min":0.749,"15min":0.749},"max":{"1min":1.6899999999999999,"5min":2.5310000000000001,"15min":2.5310000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.169,"5min":2.173,"15min":4.8449999999999998},"min":{"1min":0.88400000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.542,"5min":129.49100000000001,"15min":129.49100000000001},"last":1.155}]}]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796864,"kb_used_data":92902516,"kb_used_omap":135,"kb_used_meta":0,"kb_avail":92902516,"statfs":{"total":95948165120,"available":95132176384,"internally_reserved":0,"allocated":95132176384,"data_stored":95132176384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":139233,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68400000000000005,"15min":0.71999999999999997},"min":{"1min":0.51200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.99099999999999999,"5min":2.7130000000000001,"15min":2.7130000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.754,"15min":0.71299999999999997},"min":{"1min":0.442,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1100000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.73499999999999999,"15min":0.749},"min":{"1min":0.50800000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":1.119,"15min":1.119},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.006,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.92200000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.77900000000000003,"15min":0.74199999999999999},"min":{"1min":0.56699999999999995,"5min":0.373,"15min":0.373},"max":{"1min":1.1870000000000001,"5min":1.292,"15min":1.292},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.88600000000000001,"15min":0.86599999999999999},"min":{"1min":0.64000000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.038,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.97999999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.97999999999999998,"15min":0.97599999999999998},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.1599999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.01},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.997,"15min":1.0860000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.1970000000000001,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.1080000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1,"15min":1.0029999999999999},"min":{"1min":0.79700000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.411,"5min":2.165,"15min":2.165},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1259999999999999,"15min":1.139},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.4079999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.363}]},{"osd":6,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.137,"15min":1.1140000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.347,"5min":1.615,"15min":1.615},"last":1.268},{"interface":"front","average":{"1min":1.083,"5min":1.119,"15min":1.095},"min":{"1min":0.81799999999999995,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.327,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.0580000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:38:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.91400000000000003,"5min":0.755,"15min":0.755},"max":{"1min":1.3149999999999999,"5min":1.71,"15min":1.71},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.157,"5min":1.9399999999999999,"15min":4.0789999999999997},"min":{"1min":0.94499999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.486,"5min":80.903000000000006,"15min":80.903000000000006},"last":1.2869999999999999}]}]}],"pool_statfs":[]}} 2023-09-30T00:38:36.895 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:38:37.207 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:38:37.209 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:38:37.210 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:38:37.210 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:38:37.210 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.225 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:38:37.225 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:38:37.225 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.286 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:38:37.287 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:38:37.287 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.302 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:38:37.302 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:38:37.303 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.318 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:38:37.318 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T00:38:37.318 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.338 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T00:38:37.338 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T00:38:37.338 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.399 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T00:38:37.400 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T00:38:37.400 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.415 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T00:38:37.415 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T00:38:37.415 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.433 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T00:38:37.433 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:38:37.433 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T00:38:37.434 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.463 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T00:38:37.464 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:38:37.464 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.498 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:38:37.498 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:38:37.498 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:38:37.498 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.517 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:38:37.517 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T00:38:37.517 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.577 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T00:38:37.577 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T00:38:37.577 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:38:37.596 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T00:38:37.596 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:38:37.597 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:38:37.616 INFO:teuthology.orchestra.run.smithi110.stdout:1696033976.6282423 mgr.x (mgr.4101) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:38:37.617 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:38:37.617 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:38:37.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'must scrub before tier agent can activate' | head -n 1 2023-09-30T00:38:37.694 INFO:teuthology.orchestra.run.smithi110.stdout:1696033976.6282423 mgr.x (mgr.4101) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:38:37.694 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:38:37.695 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:38:37.924 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:38:37.924 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:38:38.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:38:38.207 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:38:38.422 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:38:38.422 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:38:38.621 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:38:38.622 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T00:38:38.831 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:38:38.831 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T00:38:39.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:38:39.045 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T00:38:39.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:38:39.222 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T00:38:39.397 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:38:39.398 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/data/mon.a.tgz 2023-09-30T00:38:39.399 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-09-30T00:38:39.405 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.0RjASJz3kw 2023-09-30T00:38:39.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0RjASJz3kw 2023-09-30T00:38:39.523 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.0RjASJz3kw 2023-09-30T00:38:39.544 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.0RjASJz3kw is 297KB 2023-09-30T00:38:39.606 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.0RjASJz3kw 2023-09-30T00:38:39.613 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/data/mon.c.tgz 2023-09-30T00:38:39.613 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-09-30T00:38:39.619 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.OkFzyytKaW 2023-09-30T00:38:39.620 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.OkFzyytKaW 2023-09-30T00:38:39.735 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.OkFzyytKaW 2023-09-30T00:38:39.757 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.OkFzyytKaW is 298KB 2023-09-30T00:38:39.818 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.OkFzyytKaW 2023-09-30T00:38:39.825 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/data/mon.b.tgz 2023-09-30T00:38:39.825 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2023-09-30T00:38:39.832 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.02Bp27v7su 2023-09-30T00:38:39.832 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.02Bp27v7su 2023-09-30T00:38:39.905 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.02Bp27v7su 2023-09-30T00:38:39.922 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.02Bp27v7su is 298KB 2023-09-30T00:38:39.985 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.02Bp27v7su 2023-09-30T00:38:39.992 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:38:39.992 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:38:39.995 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:38:40.013 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:38:40.013 DEBUG:teuthology.orchestra.run.smithi110:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:38:40.053 DEBUG:teuthology.orchestra.run.smithi157:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:38:40.077 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:38:40.085 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:38:40.091 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:38:40.092 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/remote/smithi110/crash 2023-09-30T00:38:40.094 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:38:40.108 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/remote/smithi157/crash 2023-09-30T00:38:40.109 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:38:40.129 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:38:40.129 DEBUG:teuthology.orchestra.run.smithi110:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:38:40.153 DEBUG:teuthology.orchestra.run.smithi157:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:38:40.883 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:38:40.883 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/remote/smithi110/log 2023-09-30T00:38:40.884 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:38:41.118 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/remote/smithi157/log 2023-09-30T00:38:41.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:38:41.206 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:38:41.277 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6b6156485ece430baf4cf7f370d317fe Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:38:41.279 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:38:41.300 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:38:41.308 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:38:41.308 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:38:41.320 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:38:41.384 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:38:41.393 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:38:41.393 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:38:41.396 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:38:41.416 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T00:38:41.416 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2023-09-30T00:38:41.417 INFO:teuthology.orchestra.run.smithi110.stdout:+hv01.front.sepi 44.4.53.2 2 u 32 64 377 0.089 1.323 0.642 2023-09-30T00:38:41.417 INFO:teuthology.orchestra.run.smithi110.stdout:*hv02.front.sepi 63.231.80.2 3 u 40 64 377 0.094 2.490 1.229 2023-09-30T00:38:41.417 INFO:teuthology.orchestra.run.smithi110.stdout:+hv03.front.sepi 74.6.168.72 3 u 40 64 377 0.065 -0.657 1.269 2023-09-30T00:38:41.417 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:38:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T00:38:41.419 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2023-09-30T00:38:41.419 INFO:teuthology.orchestra.run.smithi157.stdout:*hv01.front.sepi 44.4.53.2 2 u 35 64 377 0.066 1.566 0.822 2023-09-30T00:38:41.419 INFO:teuthology.orchestra.run.smithi157.stdout:+hv02.front.sepi 63.231.80.2 3 u 35 64 377 0.078 1.215 0.672 2023-09-30T00:38:41.419 INFO:teuthology.orchestra.run.smithi157.stdout:+hv03.front.sepi 74.6.168.72 3 u 33 64 377 0.072 0.765 1.153 2023-09-30T00:38:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:38:41.420 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:38:41.429 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:38:41.429 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:38:41.437 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:38:41.445 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:38:41.453 INFO:teuthology.task.internal:Duration was 846.247711 seconds 2023-09-30T00:38:41.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:38:41.462 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:38:41.462 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:38:41.465 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:38:41.514 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:38:41.515 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T00:38:41.515 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:38:41.527 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2023-09-30T00:38:41.527 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:38:41.540 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:38:41.540 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:38:41.543 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:38:41.573 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:38:41.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:38:41.617 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:38:41.681 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:38:41.691 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:38:41.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:38:41.704 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:38:41.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:38:41.742 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:38:41.745 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:38:41.757 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2023-09-30T00:38:41.777 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2023-09-30T00:38:41.805 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:38:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:38:41.841 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:38:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:38:41.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:38:41.858 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:38:41.858 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/remote/smithi110 2023-09-30T00:38:41.859 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:38:41.880 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725/remote/smithi157 2023-09-30T00:38:41.880 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:38:41.912 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:38:41.912 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:38:41.914 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:38:41.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:38:41.931 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:38:41.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:38:41.940 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:38:41.941 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:38:41.961 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:38:41.967 INFO:teuthology.orchestra.run.smithi110.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:38 /home/ubuntu/cephtest 2023-09-30T00:38:41.969 INFO:teuthology.orchestra.run.smithi157.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:38 /home/ubuntu/cephtest 2023-09-30T00:38:41.971 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:38:41.981 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:38:42.001 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:38:42.027 DEBUG:teuthology.nuke:shortname: smithi110 2023-09-30T00:38:42.028 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:38:42.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:14:35.498712', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T00:38:42.055 DEBUG:teuthology.nuke:shortname: smithi157 2023-09-30T00:38:42.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:38:42.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:14:35.500144', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T00:38:42.113 INFO:teuthology.orchestra.console.smithi110:Power off 2023-09-30T00:38:42.114 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:38:42.137 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2023-09-30T00:38:42.137 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:42.155 INFO:teuthology.orchestra.console.smithi157:Power off 2023-09-30T00:38:42.156 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:38:42.166 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-09-30T00:38:42.180 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2023-09-30T00:38:42.180 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:42.215 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T00:38:46.167 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:46.216 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:46.279 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-09-30T00:38:46.326 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T00:38:50.280 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:50.326 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:50.392 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-09-30T00:38:50.437 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-09-30T00:38:54.394 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:54.438 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:38:54.505 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2023-09-30T00:38:54.547 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-09-30T00:38:54.606 INFO:teuthology.orchestra.console.smithi110:Power off completed 2023-09-30T00:38:54.648 INFO:teuthology.orchestra.console.smithi157:Power off completed 2023-09-30T00:38:54.753 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} duration: 846.2477107048035 failure_reason: '"1696033976.6282423 mgr.x (mgr.4101) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6b6156485ece430baf4cf7f370d317fe status: fail success: false 2023-09-30T00:38:54.753 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:38:54.832 INFO:teuthology.run:FAIL